Discrete-state Continuous-time Markov Research Articles

Reliability assessment of performance-based balanced systems with rebalancing mechanisms

Performance-based balanced systems (PBSs) are widely used as critical equipment in many fields, so their reliability assessment has become a hot topic. Taking the battery pack PBS as an example, the balance means the performance differences between components are within an acceptable range, where the common bus is used to redistribute the performance for rebalancing, and the redistribution is affected by the transmission loss and transmission capacity limit. To reflect the above rebalancing mechanisms, a reliability evaluation method is proposed for PBSs with common bus performance sharing (PBSs-CBPS) considering balance degree threshold, transmission loss, and transmission capacity limit. Firstly, a continuous-time discrete-state Markov model is built to address the transition behaviors of components. Next, considering the effects of balance degree threshold and transmission loss, the rebalancing process is formulated as a nonlinear programming problem to obtain the performance of components after rebalancing, and the system reliability model is established. Then, the universal generation function (UGF) method combined with nonlinear programming is proposed to calculate system reliability, and Paper-Cut UGF Algorithm (PCUGF) is proposed to improve computational efficiency. Finally, an analytical example and a numerical example of 6S1P lithium-ion battery packs are given to show the effectiveness of the proposed method.

Multi-state Markov model analysis of disease outcomes and influencing factors in HIV infected individuals receiving antiretroviral therapy in Luzhou of Sichuan province, 2010-2021

Objective: To construct a multi-state Markov model and analyze the disease outcomes and its influencing factors in HIV infected individuals receiving antiretroviral therapy. Methods: A retrospective cohort analysis was conducted in HIV infected individuals receiving antiretroviral therapy in Luzhou of Sichuan province from 2010 to 2021. The disease status was divided into CD4+T lymphocytes (CD4) counts >500 cells/μl, 350-500 cells/μl, 200-349 cells/μl, ≤199 cells/μl and death indicated by S1-S5 in turn. A reversible continuous-time discrete-state multi-state Markov model was constructed for the analysis of disease progression features. Results: A total of 7 542 HIV infected individuals receiving antiretroviral therapy were included, and the median age (Q1, Q3) was 53.4 (41.2, 64.5) years old. The transition intensity of S3→S2 was higher. During follow-up, the transition probability of S4→S5 increased gradually. Influencing factors analysis of disease outcomes in HIV infected individuals receiving antiretroviral therapy showed that compared with individuals 15-24 years old, the transition intensities of S2→S1, S3→S2 and S4→S3 were lower and the transition intensity of S3→S4 was higher in individuals ≥45 years old. Compared with single individuals, the transition intensities of S3→S2 and S4→S3 were higher and the transition intensities of S3→S4 and S4→S5 were lower in married individuals. The transition intensity of S1→S2 was higher in individuals with baseline CD4 counts ≤500 cells/μl than in individuals with baseline CD4 counts >500 cells/μl. The transition intensity of S3→S4 in individuals diagnosed during 2011-2015 was lower than that in individuals diagnosed in 2010 and before. Conclusions: HIV infected individuals receiving antiretroviral therapy tended to shift to the previous disease status, suggesting that antiretroviral therapy was conducive to immune reconstitution. Older age (≥45 years old), being married, low baseline CD4 counts and being diagnosed in 2010 and before were the risk factors for disease progression.

Condition-based Maintenance Optimization of Degradable Systems

This paper develops a mathematical model for condition-based maintenance optimization of multi-state systems. The majority of the existing literature on maintenance optimization assume that there is no additional cost incurred because of side effects of equipment degradation. Nevertheless, as the operating cost increases with equipment age and degradation, it is important to consider the degradation side effects in the maintenance decision-making process. An important feature of the proposed model lies in the fact that it incorporates side effect of degradation process into condition-based preventive maintenance optimization. We develop a continuous-time discrete-state Markov chain model describing the deterioration stochastic process of a single component. The component is modeled as a multi-state system, where each discrete state is characterized by a degradation level. Numerical examples show the importance of considering such side effect costs when optimizing the choice of maintenance policy. The proposed model is extended to deal with multi-state series systems. Using an example of a series system with two components, it is shown that preventive maintenance and side effect costs should not be optimized for each component individually, but from the perspective of the series system as a whole.

A Markov reward process-based framework for resilience analysis of multistate energy systems under the threat of extreme events

Energy systems are increasingly exposed to the threats of extreme events like floods, earthquakes and hurricanes. In practice, the behaviors of the systems affected by these extreme events are often modeled by multistate models to facilitate the analysis. In this paper, we develop a generic framework for resilience modeling and analysis of multistate energy systems. A multistate resilience model is developed based on a Markov reward process model, where the degradation and recovery of system performance are characterized by a continuous time discrete state Markov chain and the losses caused by the extreme event is modeled by the reward rates associated with the sojourns in the degradation states and the transitions among the states. Four numerical metrics are defined to describe different aspects of system resilience, i.e. the resistant, absorption, recovery and overall resilience. A simulation-based algorithm is proposed for resilience analysis of multistate energy systems. The developed methods are applied for resilience modeling and analysis of a Nuclear Power Plant (NPP) under the threat of earthquakes. The Markov reward process model is developed following a probabilistic seismic hazard analysis, a fragility analysis and an event tree modeling of accident evolutions. Both a time-static and time-dependent resilience analysis are conducted and the results show that the developed model is able to comprehensively describe the resilience of multistate energy systems under the threats of extreme events.

Exact analysis of summary statistics for continuous-time discrete-state Markov processes on networks using graph-automorphism lumping

We propose a unified framework to represent a wide range of continuous-time discrete-state Markov processes on networks, and show how many network dynamics models in the literature can be represented in this unified framework. We show how a particular sub-set of these models, referred to here as single-vertex-transition (SVT) processes, lead to the analysis of quasi-birth-and-death (QBD) processes in the theory of continuous-time Markov chains. We illustrate how to analyse a number of summary statistics for these processes, such as absorption probabilities and first-passage times. We extend the graph-automorphism lumping approach [Kiss, Miller, Simon, Mathematics of Epidemics on Networks, 2017; Simon, Taylor, Kiss, J. Math. Bio. 62(4), 2011], by providing a matrix-oriented representation of this technique, and show how it can be applied to a very wide range of dynamical processes on networks. This approach can be used not only to solve the master equation of the system, but also to analyse the summary statistics of interest. We also show the interplay between the graph-automorphism lumping approach and the QBD structures when dealing with SVT processes. Finally, we illustrate our theoretical results with examples from the areas of opinion dynamics and mathematical epidemiology.

Reliability-Based Design Optimization of Load Sharing Systems Using SSI-Markov Models

This paper presents a novel single loop approach to design the components of the load sharing systems by optimally allocating the failure probabilities to each component, thereby satisfying the overall system reliability requirement. The Reliability–Based Design Optimization (RBDO) of load sharing systems is computationally intensive due to the dynamic nature of component failure probabilities, since the failure of one component will vary the failure probabilities of other working components. Many RBDO methods have been successfully utilized to design individual components, however using these methods for handling system level reliability constraints is still a challenging task. This is because of a drop in accuracy and computational efficiency, especially when considering a load sharing system, where there is dependency in failure probabilities of components. The key idea is to integrate Stress–Strength Interference (SSI) theory with discrete (or) continuous time-discrete state Markov model for the reliability assessment of system, with the states being the condition of components (working/failed). This method takes advantage of the state transition probability matrix to represent the dynamic nature of the system performance. A numerical example of a simple load sharing system with two I-Beams is presented to illustrate and evaluate the performance of the proposed methodology.

Slow-scale split-step tau-leap method for stiff stochastic chemical systems

Tau-leaping is a family of algorithms for the approximate simulation of the discrete state continuous time Markov chains. Motivation for the development of such methods can be found, for instance, in the fields of chemical kinetics and systems biology. It is known that the dynamical behavior of biochemical systems is often intrinsically stiff representing a serious challenge for their numerical approximation. The naive extension of stiff deterministic solvers to stochastic integration often yields numerical solutions with either impractically large relaxation times or incorrectly resolved covariance. In this paper, we propose a splitting heuristic which helps to resolve some of these issues. The proposed integrator contains a number of unknown parameters which are estimated for each particular problem from the moment equations of the corresponding linearized system. We show that this method is able to reproduce the exact mean and variance of the linear scalar test equation and demonstrates a good accuracy for the arbitrarily stiff systems at least in the linear case. The numerical examples for both linear and nonlinear systems are also provided and the obtained results confirm the efficiency of the considered splitting approach.

Continuous-time reinforcement learning approach for portfolio management with time penalization

This paper considers the problem of policy optimization in the context of continuous-time Reinforcement Learning (RL), a branch of artificial intelligence, for financial portfolio management purposes. The underlying asset portfolio process is assumed to possess a continuous-time discrete-state Markov chain structure involving the simplex and ergodicity constraints. The goal of the portfolio problem is the redistribution of a fund into different financial assets. One general assumption has to be set, namely that the market is arbitrage-free (no price arbitrage is possible) then the problem of how to obtain the optimal policy is solvable. We provide a RL solution based on an actor/critic architecture in which the market is characterized by a restriction called transaction cost, involving time penalization. The portfolio problem in Markov chains is determined by solving a convex quadratic minimization problem with linear constraints. Any Markov chain is generated by a stochastic transition matrices and the mathematical expectations of the rewards. In particular, we estimate the elements of the transition rate matrices and the mathematical expectations of the rewards. This method learns the optimal strategy in order to make a decision on what portfolio weight to take for a single period. With this strategy, the agent is able to choose the state with maximum utility and select its respective action. The optimal policy computation is solved employing a proximal optimization novel approach, which involves time penalization in the transaction costs and the rewards. We employ the Lagrange multipliers approach to include the restrictions of the market and those that are imposed by the continuous time frame. Moreover, a specific numerical example in baking, that fit into the general framework of portfolio, validates the effectiveness and usefulness of the proposed method.

Stochastic stability of neutral-type Markovian-jumping BAM neural networks with time varying delays

In this paper, stochastic stability of neutral type Markovian-jumping bidirectional associative memory (BAM) neural networks is investigated. The jumping parameters are modeled as a continuous-time discrete-state Markov chain. The activation functions are supposed to be bounded and globally Lipschitz continuous. Furthermore, based on the Lyapunov–Krasovskii functional, a linear matrix inequality (LMI) approach is developed to establish sufficient conditions and novel delay-dependent conditions are established for the stochastic asymptotic stability of Markovian jumping BAM neural networks. The condition is presented in terms of linear matrix inequalities (LMIs), which can be easily checked by using MATLAB LMI toolbox. Finally, numerical examples are provided to show the effectiveness of the main results.

Continuous-time mean variance portfolio with transaction costs: a proximal approach involving time penalization

ABSTRACTThis paper proposes a new continuous-time optimization solution that enables the computation of the portfolio problem (based on the utility option pricing and the shortfall risk minimization). We first propose a dynamical stock price process, and then, we transform the solution to a continuous-time discrete-state Markov decision processes. The market behavior is characterized by considering arbitrage-free and assessing transaction costs. To solve the problem, we present a proximal optimization approach, which considers time penalization in the transaction costs and the utility. In order to include the restrictions of the market, as well as those that imposed by the continuous-time space, we employ the Lagrange multipliers approach. As a result, we obtain two different equations: one for computing the portfolio strategies and the other for computing the Lagrange multipliers. Each equation in the portfolio is an optimization problem, for which the necessary condition of a maximum/minimum is solved employing the gradient method approach. At each step of the iterative proximal method, the functional increases and finally converges to a final portfolio. We show the convergence of the method. A numerical example showing the effectiveness of the proposed approach is also developed and presented.

A scalable moment-closure approximation for large-scale biochemical reaction networks.

MotivationStochastic molecular processes are a leading cause of cell-to-cell variability. Their dynamics are often described by continuous-time discrete-state Markov chains and simulated using stochastic simulation algorithms. As these stochastic simulations are computationally demanding, ordinary differential equation models for the dynamics of the statistical moments have been developed. The number of state variables of these approximating models, however, grows at least quadratically with the number of biochemical species. This limits their application to small- and medium-sized processes.ResultsIn this article, we present a scalable moment-closure approximation (sMA) for the simulation of statistical moments of large-scale stochastic processes. The sMA exploits the structure of the biochemical reaction network to reduce the covariance matrix. We prove that sMA yields approximating models whose number of state variables depends predominantly on local properties, i.e. the average node degree of the reaction network, instead of the overall network size. The resulting complexity reduction is assessed by studying a range of medium- and large-scale biochemical reaction networks. To evaluate the approximation accuracy and the improvement in computational efficiency, we study models for JAK2/STAT5 signalling and NFκB signalling. Our method is applicable to generic biochemical reaction networks and we provide an implementation, including an SBML interface, which renders the sMA easily accessible.Availability and implementationThe sMA is implemented in the open-source MATLAB toolbox CERENA and is available from https://github.com/CERENADevelopers/CERENA.Supplementary information Supplementary data are available at Bioinformatics online.

The Liquidity Regimes and the Prepayment Option of a Corporate Loan in the Finite Horizon Case

We investigate the prepayment option related to a corporate loan. The default intensity of the firm is supposed to follow a Cox-Ingersoll-Ross (CIR) process and the short interest rate is assumed constant. A liquidity term that represents the funding costs of the bank is introduced and modeled as a continuous time discrete state Markov jump process. The prepayment option is an American option with the payoff being an implicit function of the parameters of the problem. We give a verification result that allows to compute the price of the option. Numerical results are completely consistent with the theory; it is seen that the exercise domain may entirely disappear during such a liquidity crisis meaning that it is not optimal for the borrower to prepay. The method allows to quantify and interpret these findings.

A master equation and moment approach for biochemical systems with creation-time-dependent bimolecular rate functions.

Noise and stochasticity are fundamental to biology and derive from the very nature of biochemical reactions where thermal motion of molecules translates into randomness in the sequence and timing of reactions. This randomness leads to cell-to-cell variability even in clonal populations. Stochastic biochemical networks have been traditionally modeled as continuous-time discrete-state Markov processes whose probability density functions evolve according to a chemical master equation (CME). In diffusion reaction systems on membranes, the Markov formalism, which assumes constant reaction propensities is not directly appropriate. This is because the instantaneous propensity for a diffusion reaction to occur depends on the creation times of the molecules involved. In this work, we develop a chemical master equation for systems of this type. While this new CME is computationally intractable, we make rational dimensional reductions to form an approximate equation, whose moments are also derived and are shown to yield efficient, accurate results. This new framework forms a more general approach than the Markov CME and expands upon the realm of possible stochastic biochemical systems that can be efficiently modeled.

PREPAYMENT OPTION OF A PERPETUAL CORPORATE LOAN: THE IMPACT OF THE FUNDING COSTS

We investigate in this paper a perpetual prepayment option related to a corporate loan. The short interest rate and default intensity of the firm are supposed to follow Cox–Ingersoll–Ross (CIR) processes. A liquidity term that represents the funding costs of the bank is introduced and modeled as a continuous time discrete state Markov chain. The prepayment option needs specific attention as the payoff itself is a derivative product and thus an implicit function of the parameters of the problem and of the dynamics. We prove verification results that allows to certify the geometry of the exercise region and compute the price of the option. We show moreover that the price is the solution of a constrained minimization problem and propose a numerical algorithm building on this result. The algorithm is implemented in a two-dimensional code and several examples are considered. It is found that the impact of the prepayment option on the loan value is not to be neglected and should be used to assess the risks related to client prepayment. Moreover, the Markov chain liquidity model is seen to describe more accurately clients' prepayment behavior than a model with constant liquidity.

A score-test on measurement errors in rating transition times

We model credit rating histories as continuous-time discrete-state Markov processes. Infrequent monitoring of the debtors’ solvency will result in erroneous observations of the rating transition times, and consequently in biased parameter estimates. We develop a score test against such measurement errors in the transition data that is independent of the error distribution. We derive the asymptotic χ2-distribution for the test statistic under the null by stochastic limit theory. The test is applied to an international corporate portfolio, while accounting for economic and debtor-specific covariates. The test indicates that measurement errors in the transition times are a real problem in practice.

Modelling and analysis of a multi-stage system involving batch processors with incompatible job families

Batch processors can concurrently process more than one job, and are commonly used in complex manufacturing systems such as wafer fabrication facilities (wafer fabs). Previous work has shown that there can be a significant reduction in cycle time when the control of the upstream serial processor is dependent on the batch processor. We wish to determine whether this is also true when the upstream processor is also a batch processor, with smaller capacity. We model a two-stage system with an upstream batch processor and a downstream batch processor under differing simple control policies as a continuous time-discrete state Markov chain. In doing so, we show that the concept of constraining the production of the upstream processor in accordance to the anticipated needs of the downstream batch processor can similarly reduce mean system cycle time when the upstream processor is a smaller batch processor. We also perform simulation experiments, using parameters close to those found in wafer fabs, and confirm that the strategy of prioritising the formation of larger batches for the downstream batch processor typically results in substantial cycle time reductions for the two-stage system.

CO oxidation on Ir(111) surfaces under large non-Gaussian noise

In this article we consider the CO oxidation on Ir(111) surfaces under large external noise with large autocorrelation imposed on the composition of the feed gas, both in experiments and in theory. We report new experimental results that show how the fluctuations force the reaction rate to jump between two well defined states. The statistics of the reaction rate depend on those of the external noise, and neither of them have a gaussian distribution, and thus they cannot be modeled by white or colored noise. A continuous-time discrete-state Markov process is proposed as a suitable model for the observed phenomena. The model captures the main features of the observed fluctuations and can be modified to accommodate other surface reactions and other systems under non-gaussian external noise.

Stability analysis of Markovian jump stochastic BAM neural networks with impulse control and mixed time delays.

This paper discusses the issue of stability analysis for a class of impulsive stochastic bidirectional associative memory neural networks with both Markovian jump parameters and mixed time delays. The jumping parameters are modeled as a continuous-time discrete-state Markov chain. Based on a novel Lyapunov-Krasovskii functional, the generalized Itô's formula, mathematical induction, and stochastic analysis theory, a linear matrix inequality approach is developed to derive some novel sufficient conditions that guarantee the exponential stability in the mean square of the equilibrium point. At the same time, we also investigate the robustly exponential stability in the mean square of the corresponding system with unknown parameters. It should be mentioned that our stability results are delay-dependent, which depend on not only the upper bounds of time delays but also their lower bounds. Moreover, the derivatives of time delays are not necessarily zero or smaller than one since several free matrices are introduced in our results. Consequently, the results obtained in this paper are not only less conservative but also generalize and improve many earlier results. Finally, two numerical examples and their simulations are provided to show the effectiveness of the theoretical results.

Evaluating the performance of look-ahead policies for upstream serial processor with downstream batch processor serving incompatible job families and finite buffer sizes

Batch processors can concurrently process more than one job. In wafer fabrication, the processing time of a batch is independent of batch size, and only jobs from the same job family can be batched together. We consider a two-stage subsystem of a wafer fabrication facility (wafer fab), comprised of the diffusion furnace (a batch processor) and its upstream serial processor, with random job arrivals. We hypothesise that allowing the serial processor to anticipate the job family preference of the batch processor will reduce the overall cycle time of jobs passing through this system. To evaluate this hypothesis, we model the performance of the two-stage system under different system parameters and processor control policies as discrete state continuous time Markov chains. We characterise the system performance and show that the concept of constraining the production of the upstream processor according to the anticipated needs of the batch processor can reduce the mean cycle time of jobs being processed. We also perform simulation experiments to show that a simple heuristic based on this insight can translate to substantial cycle time reductions for systems with assumptions closer to those found in wafer fabs.

A data-integrated method for analyzing stochastic biochemical networks.

Variability and fluctuations among genetically identical cells under uniform experimental conditions stem from the stochastic nature of biochemical reactions. Understanding network function for endogenous biological systems or designing robust synthetic genetic circuits requires accounting for and analyzing this variability. Stochasticity in biological networks is usually represented using a continuous-time discrete-state Markov formalism, where the chemical master equation (CME) and its kinetic Monte Carlo equivalent, the stochastic simulation algorithm (SSA), are used. These two representations are computationally intractable for many realistic biological problems. Fitting parameters in the context of these stochastic models is particularly challenging and has not been accomplished for any but very simple systems. In this work, we propose that moment equations derived from the CME, when treated appropriately in terms of higher order moment contributions, represent a computationally efficient framework for estimating the kinetic rate constants of stochastic network models and subsequent analysis of their dynamics. To do so, we present a practical data-derived moment closure method for these equations. In contrast to previous work, this method does not rely on any assumptions about the shape of the stochastic distributions or a functional relationship among their moments. We use this method to analyze a stochastic model of a biological oscillator and demonstrate its accuracy through excellent agreement with CME/SSA calculations. By coupling this moment-closure method with a parameter search procedure, we further demonstrate how a model's kinetic parameters can be iteratively determined in order to fit measured distribution data.