Continuous-time Markov Chain Research Articles

Modeling the delay propagation dynamics in high-speed railway networks with a traffic-driven SIS epidemic model

The propagation of delays is a prevalent issue in high-speed railway (HSR) networks. However, the underlying properties of delay propagation have not been fully explored, hindering the effective management of this problem. In this paper, considering the impact of traffic flow, we model the delay propagation as a traffic-driven susceptible–infected–susceptible (SIS) epidemic spreading process. We introduce a traffic frequency matrix to estimate the traffic rate and further derive the propagation dynamic equations using a continuous-time Markov chain method. We then derive the epidemic threshold of the model, which is proportional to the recovery rate and inversely proportional to the average path length. Finally, through simulation, we study the impacts of various factors including the infection rate, recovery rate, and the degree and number of initial delay stations. Our work provides some insights for the prediction and control of delay propagation in HSR networks.

Optimal Inspection and Maintenance Policy: Integrating a Continuous-Time Markov Chain into a Homing Problem

The state of a machine is modeled as a controlled continuous-time Markov chain. Moreover, the machine is being serviced at random times. The aim is to maximize the time until the machine must be repaired, while taking the maintenance costs into account. The dynamic programming equation satisfied by the value function is derived, enabling optimal decision-making regarding inspection rates, and special problems are solved explicitly. This approach minimizes direct maintenance costs along with potential failure expenses, establishing a robust methodology for determining inspection frequencies in reliability-centered maintenance. The results contribute to the advancement of maintenance strategies and provide explicit solutions for particular cases, offering ideas for application in reliability engineering.

Review of Continuous Time Markov Bridges: Models, Algorithms and Applications

Abstract. Continuous time Markov chains (CTMC) with start point and endpoint, i.e., a continuous time Markov Bridge, have become an important mathematical model in a wide variety of scientific disciplines and simulations. In this research, we review some classical Markov Bridge simulation algorithms and Markov Bridge's mathematical model, Kinetic Monte Carlo, Direct Sampling, Reject Sampling and Uniform Sampling with a potential application algorithm in biological protein folding evolution problem.

A general valuation framework for rough stochastic local volatility models and applications

Rough volatility models are a new class of stochastic volatility models that have been shown to provide a consistently good fit to implied volatility smiles of SPX options. They are continuous-time stochastic volatility models, whose volatility process is driven by a fractional Brownian motion with the corresponding Hurst parameter less than a half. Albeit the empirical success, the valuation of derivative securities under rough volatility models is challenging. The reason is that it is neither a semi-martingale nor a Markov process. This paper proposes a novel valuation framework for rough stochastic local volatility (RSLV) models. In particular, we introduce the perturbed stochastic local volatility (PSLV) model as the semi-martingale approximation for the RSLV model and establish its existence, uniqueness, Markovian representation and convergence. Then we propose a fast continuous-time Markov chain (CTMC) approximation algorithm to the PSLV model and establish its convergence. Numerical experiments demonstrate the convergence of our approximation method to the true prices, and also the remarkable accuracy and efficiency of the method in pricing European, barrier and American options. Comparing with existing literature, a significant reduction in the CPU time to arrive at the same level of accuracy is observed.

Standby component replacement strategy for a balanced system with a standby pool

In this paper, a new rebalancing strategy is proposed by replacing the standby components for a balanced system with a standby pool. The working components are connected in series. All standby components are stored in a standby component pool. When the working components do not fail and the balance degree of the system does not exceed a threshold, the system can operate normally. The balance degree of the system is defined as the maximum state difference of all the working components. If the system becomes unbalanced or fails, a qualified standby component in the standby pool should be selected to replace the working component. The replacement strategy is given in detail. This paper uses the discrete-state continuous-time Markov process to derive the system reliability. Finally, a numerical example and sensitivity analysis are given. Considering the downtime cost and the cost of purchasing standby components, the number of standby components in the standby pool is optimized.

Modeling Emergency Traffic Using a Continuous-Time Markov Chain

This paper aims to propose a novel call for help traffic (SOS) and study its impact over Machine-to-Machine (M2M) and Human-to-Human (H2H) traffic in Internet of Things environments, specifically during disaster events. During such events (e.g., the spread COVID-19), SOS traffic, with its predicted exponential increase, will significantly influence all mobile networks. SOS traffic tends to cause many congestion overload problems that significantly affect the performance of M2M and H2H traffic. In our project, we developed a new Continuous-Time Markov Chain (CTMC) model to analyze and measure radio access performance in terms of massive SOS traffic that influences M2M and H2H traffic. Afterwards, we validate the proposed CTMC model through extensive Monte Carlo simulations. By analyzing the traffic during an emergency case, we can spot a huge impact over the three traffic types of M2M, H2H and SOS traffic. To solve the congestion problems while keeping the SOS traffic without any influence, we propose to grant the SOS traffic the highest priority over the M2M and H2H traffic. However, by implementing this solution in different proposed scenarios, the system becomes able to serve all SOS requests, while only 20% of M2M and H2H traffic could be served in the worst-case scenario. Consequently, we can alleviate the expected shortage of SOS requests during critical events, which might save many humans and rescue them from being isolated.

A stochastic analysis of the dynamic space utilization in warehouses

Improving the volume space utilization of modern warehouses is a critical objective in warehouse layout design, making the accurate estimation of space utilization across different layouts essential. However, the dynamic demand and replenishment processes of heterogeneous stock keeping units (SKUs) in modern warehouses present significant challenges in accurately assessing space utilization. Traditional models have relied on deterministic frameworks that assume constant demand and production processes, which do not fully capture the heterogeneity and stochasticity in demand arrivals, sizes, and replenishment times encountered in practical scenarios. This study aims to develop a framework for computing volume space utilization in modern block stacking warehouses, incorporating both heterogeneity and stochasticity. We investigate two models in warehouse scenarios: the lost sale model, where excess demand is lost, and the backorder model, where unmet demand is backlogged. For each model, we derive the closed-form expressions of three key types of space wastes that characterize the space utilization of warehouses: honeycombing, aisle, and top-of-lane space wastes, using a continuous-time Markov chain analytical framework. Utilizing these expressions, we analyze the tradeoffs among these wastes and identify the optimal lane depth that maximizes space utilization. Our case studies show that, in a stochastic environment, the proposed computational framework allows for a more accurate estimation of space utilization, and the application of the obtained optimal lane depth can achieve significantly higher space utilization than deterministic methods found in the literature. This underscores the importance of incorporating stochasticity and heterogeneity of demand and replenishment in layout design. Additionally, by investigating the sensitivity of space utilization to varying demand processes and replenishment strategies, we provide managerial insights for adapting warehouse layout designs in response to changes in SKUs' demand and replenishment patterns.

Predicting the Impact of Distributed Denial of Service (DDoS) Attacks in Long-Term Evolution for Machine (LTE-M) Networks Using a Continuous-Time Markov Chain (CTMC) Model

The way we connect with the physical world has completely changed because of the advancement of the Internet of Things (IoT). However, there are several difficulties associated with this change. A significant advancement has been the emergence of intelligent machines that are able to gather data for analysis and decision-making. In terms of IoT security, we are seeing a sharp increase in hacker activities worldwide. Botnets are more common now in many countries, and such attacks are very difficult to counter. In this context, Distributed Denial of Service (DDoS) attacks pose a significant threat to the availability and integrity of online services. In this paper, we developed a predictive model called Markov Detection and Prediction (MDP) using a Continuous-Time Markov Chain (CTMC) to identify and preemptively mitigate DDoS attacks. The MDP model helps in studying, analyzing, and predicting DDoS attacks in Long-Term Evolution for Machine (LTE-M) networks and IoT environments. The results show that using our MDP model, the system is able to differentiate between Authentic, Suspicious, and Malicious traffic. Additionally, we are able to predict the system behavior when facing different DDoS attacks.

Information Freshness in Service Queues With Strategic Customers

We examine a scenario in which a service provider updates its queue length information to customers at a specific frequency. Customers decide whether to join the queue based on the most recent update. Our objective is to determine the optimal updating frequency that benefits both the service provider and customers. Using a two-dimensional continuous-time Markov process, we model the actual and announced queue length processes. By proving the identical distributions of these two processes under a Poisson updating scheme, we derive closed-form solutions for customers’ utility functions. Our findings demonstrate that customers adopt a generalized mixed-threshold strategy at equilibrium, and their certainty about whether to join the queue or balk does not always increase with fresher information. Furthermore, we reveal that due to customers’ different sensitivities to information freshness, system performance metrics such as throughput and total customer utility exhibit non-monotonic behavior in response to the updating frequency. Consequently, providing fresher information does not necessarily lead to improved system performance. To address this, we propose algorithms to determine the optimal updating frequency for each system performance metric and identify the conditions under which different updating frequencies are optimal. We demonstrate that any positive updating frequency can achieve customer utility no worse than the no-information system. Additionally, in systems with high customer arrival rates, updating with a positive updating frequency can improve throughput. Furthermore, we prove conditions under which a positive and finite updating frequency can yield higher throughput and total customer utility compared to real-time information systems.

Proof-of-Trusted-Work: A lightweight blockchain consensus for decentralized IoT networks

Traditional Internet of Things (IoT) architectures that rely on centralized servers for data management and decision-making are vulnerable to security threats and privacy leakage. To address this issue, blockchain has been advocated for decentralized data management in a tamper-resistance, traceable, and transparent manner. However, a major issue that hinders the integration of blockchain and IoT lies in that, it is rather challenging for resource-constrained IoT devices to perform computation-intensive blockchain consensuses such as Proof-of-Work (PoW). Furthermore, the incentive mechanism of PoW pushes lightweight IoT nodes to aggregate their computing power to increase the possibility of successful block generation. Nevertheless, this eventually leads to the formation of computing power alliances, and significantly compromises the decentralization and security of BlockChain-aided IoT (BC-IoT) networks. To cope with these issues, we propose a lightweight consensus protocol for BC-IoT, called Proof-of-Trusted-Work (PoTW). The goal of the proposed consensus is to disincentivize the centralization of computing power and encourage the independent participation of lightweight IoT nodes in blockchain consensus. First, we put forth an on-chain reputation evaluation rule and a reputation chain for PoTW to enable the verifiability and traceability of nodes' reputations based on their contributions of computing power to the blockchain consensus, and we incorporate the multi-level block generation difficulty as a rewards for nodes to accumulate reputations. Second, we model the block generation process of PoTW and analyze the block throughput using the continuous time Markov chain. Additionally, we define and optimize the relative throughput gain to quantify and maximize the capability of PoTW that suppresses the computing power centralization (i.e., centralization suppression). Furthermore, we investigate the impact of the computing power of the computing power alliance and the levels of block generation difficulty on the centralization suppression capability of PoTW. Finally, simulation results demonstrate the consistency of the analytical results in terms of block throughput. In particular, the results show that PoTW effectively reduces the block generation proportion of the computing power alliance compared with PoW, while simultaneously improving that of individual lightweight nodes. This indicates that PoTW is capable of suppressing the centralization of computing power to a certain degree. Moreover, as the levels of block generation difficulty in PoTW increase, its centralization suppression capability strengthens.

Vulnerable options with regime switching and stochastic liquidity

Investigating default risk in pricing options holds significant practical importance, as nearly all market participants and institutions face credit risk. Additionally, economic cycles and asset liquidity are crucial factors that should be incorporated. This paper considers these factors and derives an analytical pricing formula. Specifically, we model the economic cycles through switching volatility driven by a continuous-time Markov chain, while we adopt a discounting factor based on market liquidity levels to model the asset liquidity. We establish a risk-neutral measure after embracing a regime-switching Esscher transform, and formulate a price representation to value vulnerable options analytically despite the complexity of the developed model. We conduct several numerical experiments to validate the model’s efficacy and flexibility.

Analytical computation of conditional moments in the extended Cox–Ingersoll–Ross process with regime switching: Hybrid PDE system solutions with financial applications

In this paper, we introduce a novel analytical approach for the computation of the nth conditional moments of an m-state regime-switching extended Cox–Ingersoll–Ross process driven by a continuous-time finite-state irreducible Markov chain. This approach is applicable for all integers n≥1 and m≥1, thereby ensuring wide-ranging utility. The key of our investigation is a complex hybrid system of inter-connected PDEs, derived through a utilization of the Feynman–Kac formula for regime-switching diffusion processes. Our exploration into the solutions of this hybrid PDE system culminates in the derivation of exact closed-form formulas for the conditional moments for diverse values of n and m. Additionally, we study the asymptotic characteristics of the first conditional moments for the 2-state regime-switching Cox–Ingersoll–Ross process, particularly focusing on the effects of the symmetry inherent in the Markov chain’s intensity matrix and the implications of various parameter configurations. Highlighting the practicality of our methodology, we conduct Monte Carlo simulations to not only corroborate the accuracy and computational efficacy of our proposed approach but also to demonstrate its applicability to real-world applications in financial markets. A principal application highlighted in our study is the valuation of VIX futures and VIX options within a dynamic, mean-reverting, hybrid regime-switching framework. This exemplifies the potential of our analytical method to significantly impact contemporary financial modeling and derivative pricing.

A multi-valued decision diagrams-based method for reliability analysis of performance-sharing k-out-of-n: G system considering component degradation

This paper models the reliability of a performance-sharing k-out-of-n: G system with heterogeneous degrading components and a performance-redistributing common bus. Each component may behave at various performance levels to meet its random demand. If one component exhibits performance beyond its demand, the redundant performance is redistributed to components with deficit performance via the common bus with limited capacity. The system fails if the number of operating components is less than k after sharing the redundant performance. A new analytical method based on multi-valued decision diagrams (MDDs) is put forward, which comprises an efficient model generation algorithm leveraging top-down simplification rules and a new ordering heuristic for improving MDD generation efficiency. The MDD evaluation engages the continuous-time Markov Chains to compute the steady-state probabilities of system components considering the degradation effects. Case studies of a wind power generation system and a data processing system as well as benchmark studies are conducted to illustrate the applicability and efficiency of the proposed method. A comparative study with the universal generating function-based method is also provided to further demonstrate the efficiency of the proposed MDD-based method.

Reliability and Sensitivity Analysis of Wireless Sensor Network Using a Continuous-Time Markov Process

A remarkably high growth has been observed in the uses of wireless sensor networks (WSNs), due to their momentous potential in various applications, namely the health sector, smart agriculture, safety systems, environmental monitoring, military operations, and many more. It is quite important that a WSN must have high reliability along with the least MTTF. This paper introduces a continuous-time Markov process, which is a special case of stochastic process, based on modeling of a wireless sensor network for analyzing the various reliability indices of the same. The modeling has been conducted by considering the different components, including the sensing unit, transceiver, microcontroller, power supply, standby power supply unit, and their failures/repairs, which may occur during their functioning. The study uncovered different important assessment parameters like reliability, components-wise reliability, MTTF, and sensitivity analysis. The critical components of a WSN are identified by incorporating the concept of sensitivity analysis. The outcomes emphasize that the proposed model will be ideal for understanding different reliability indices of WSNs and guiding researchers and potential users in developing a more robust wireless sensor network system.

Risk conscious investment

The risk conscious investor is defined as the maximizer of a conservative valuation or a dynamic nonlinear expectation. Both the static and dynamic problems are addressed using distortions of tail probabilities or distortions of tail measures. The multivariate static problem is solved in the context of the multivariate bilateral gamma model. For the dynamic model states and their transitions are modeled in two ways. The first defines states by a parametric model for the return distribution with transitions described by a continuous-time finite state Markov chain between the distributional possibilities. In the second model states are represented by the first four power variations with transitions given by (OU) equations for modeling stochastically the first four power variations as Tempered Fractional Lévy Processes (TFLP). Numerical solutions for policy functions are implemented in trading 768 equity assets over seven years ending December 2021.

Beyond Time-Homogeneity for Continuous-Time Multistate Markov Models

–Multistate Markov models are a canonical parametric approach for data modeling of observed or latent stochastic processes supported on a finite state space. Continuous-time Markov processes describe data that are observed irregularly over time, as is often the case in longitudinal medical data, for example. Assuming that a continuous-time Markov process is time-homogeneous, a closed-form likelihood function can be derived from the Kolmogorov forward equations—a system of differential equations with a well-known matrix-exponential solution. Unfortunately, however, the forward equations do not admit an analytical solution for continuous-time, time- inhomogeneous Markov processes, and so researchers and practitioners often make the simplifying assumption that the process is piecewise time-homogeneous. In this article, we provide intuitions and illustrations of the potential biases for parameter estimation that may ensue in the more realistic scenario that the piecewise-homogeneous assumption is violated, and we advocate for a solution for likelihood computation in a truly time-inhomogeneous fashion. Particular focus is afforded to the context of multistate Markov models that allow for state label misclassifications, which applies more broadly to hidden Markov models (HMMs), and Bayesian computations bypass the necessity for computationally demanding numerical gradient approximations for obtaining maximum likelihood estimates (MLEs). Supplemental materials are available online.

A Weak MLMC Scheme for Lévy-Copula-Driven SDEs with Applications to the Pricing of Credit, Equity and Interest Rate Derivatives

This paper develops a novel weak multilevel Monte-Carlo (MLMC) approximation scheme for Lévy-driven Stochastic Differential Equations (SDEs). The scheme is based on the state space discretization (via a continuous-time Markov chain approximation [Mijatović, Aleksandar, Matija Vidmar, and Saul Jacka. 2012. “Markov Chain Approximations for Transition Densities of Lévy Processes.” Electronic Journal of Probability 19]) of the pure-jump component of the driving Lévy process and is particularly suited if the multidimensional driver is given by a Lévy copula. The multilevel version of the algorithm requires a new coupling of the approximate Lévy drivers in the consecutive levels of the scheme, which is defined via a coupling of the corresponding Poisson point processes. The multilevel scheme is weak in the sense that the bound on the level variances is based on the coupling alone without requiring strong convergence. Moreover, the coupling is natural for the proposed discretization of jumps and is easy to simulate. The approximation scheme and its multilevel analogous are applied to examples taken from mathematical finance, including the pricing of credit, equity and interest rate derivatives.

Closed-form approximations of moments and densities of continuous–time Markov models

This paper develops power series expansions of a general class of moment functions, including transition densities and option prices, of continuous-time Markov processes, including jump–diffusions. The proposed expansions extend the ones in Kristensen and Mele (2011) to cover general Markov processes, and nest transition density and option price expansions recently developed in the literature, thereby connecting seemingly different ideas in a unified framework. We show how the general expansion can be implemented for fully general jump–diffusion models. We provide a new theory for the validity of the expansions which shows that series expansions are not guaranteed to converge as more terms are added in general once the time span of interest gets larger than some model–specific threshold. Thus, these methods should be used with caution when applied to problems with a larger time span of interest, such as long-term options or data observed at a low frequency. At the same time, the numerical studies in this paper demonstrate good performance of the proposed implementation in practice when applied to pricing options with time to maturity below three months. Thus, our expansions are particularly well suited for pricing ultra-short-term (such as “zero–day”) options.

Performing global sensitivity analysis on simulations of a continuous-time Markov chain model motivated by epidemiology

Performing global sensitivity analysis on simulations of a continuous-time Markov chain model motivated by epidemiology

A Bisimulation-Based Foundation for Scale Reductions of Continuous-Time Markov Chains

A Bisimulation-Based Foundation for Scale Reductions of Continuous-Time Markov Chains