Finite-state Markov Chain Research Articles

Analytical computation of conditional moments in the extended Cox–Ingersoll–Ross process with regime switching: Hybrid PDE system solutions with financial applications

In this paper, we introduce a novel analytical approach for the computation of the nth conditional moments of an m-state regime-switching extended Cox–Ingersoll–Ross process driven by a continuous-time finite-state irreducible Markov chain. This approach is applicable for all integers n≥1 and m≥1, thereby ensuring wide-ranging utility. The key of our investigation is a complex hybrid system of inter-connected PDEs, derived through a utilization of the Feynman–Kac formula for regime-switching diffusion processes. Our exploration into the solutions of this hybrid PDE system culminates in the derivation of exact closed-form formulas for the conditional moments for diverse values of n and m. Additionally, we study the asymptotic characteristics of the first conditional moments for the 2-state regime-switching Cox–Ingersoll–Ross process, particularly focusing on the effects of the symmetry inherent in the Markov chain’s intensity matrix and the implications of various parameter configurations. Highlighting the practicality of our methodology, we conduct Monte Carlo simulations to not only corroborate the accuracy and computational efficacy of our proposed approach but also to demonstrate its applicability to real-world applications in financial markets. A principal application highlighted in our study is the valuation of VIX futures and VIX options within a dynamic, mean-reverting, hybrid regime-switching framework. This exemplifies the potential of our analytical method to significantly impact contemporary financial modeling and derivative pricing.

Risk conscious investment

The risk conscious investor is defined as the maximizer of a conservative valuation or a dynamic nonlinear expectation. Both the static and dynamic problems are addressed using distortions of tail probabilities or distortions of tail measures. The multivariate static problem is solved in the context of the multivariate bilateral gamma model. For the dynamic model states and their transitions are modeled in two ways. The first defines states by a parametric model for the return distribution with transitions described by a continuous-time finite state Markov chain between the distributional possibilities. In the second model states are represented by the first four power variations with transitions given by (OU) equations for modeling stochastically the first four power variations as Tempered Fractional Lévy Processes (TFLP). Numerical solutions for policy functions are implemented in trading 768 equity assets over seven years ending December 2021.

Political institutions and output collapses

This paper examines whether major output collapses are more likely under autocracy. Using data on 123 developing countries over 1971–2016, we model the joint evolution of output growth and political institutions as a finite state Markov chain with a two-dimensional state space. We study how countries move between states. We find that growth is more likely to be sustained under democracy than under autocracy; output collapses are more likely to deepen under autocracy; and stagnation under autocracy can give way to outright collapse. Democratic countries appear to be more resilient.

Metastable Γ-expansion of finite state Markov chains level two large deviations rate functions

Metastable Γ-expansion of finite state Markov chains level two large deviations rate functions

ROLE OF INTERVENTION STRATEGIES AND PSYCHOLOGICAL EFFECT ON THE CONTROL OF INFECTIOUS DISEASES IN THE RANDOM ENVIRONMENT

Infectious disease outbreaks have caused significant losses to populations worldwide in terms of morbidity, social and economic costs. This has prompted governments to seek suitable intervention policies to suppress the outbreaks. This paper aims to explore the intrinsic impact of intervention strategies and the psychological behavior of people on the spread of infectious diseases. Employing the finite state Markov chain, we formulate a new stochastic SIS epidemic model under regime switching by using the Beddington–DeAngelis transmission function, in the presence of some governmental control measures. The study of the model is centered on three different scenarios in terms of varying two control parameters. We analyze the deterministic system for the global dynamics in terms of basic reproductive number. Some new threshold parameters are defined to discuss the extinction and the persistence of the disease in the stochastic setting. Numerical assessments were undertaken to validate theoretical findings for the deterministic and stochastic systems. Our findings strongly suggest that heightened awareness among susceptible individuals contributes significantly to the eradication of infection and fosters a stable epidemiological situation. As levels of public awareness increase, particularly within the susceptible population, a discernible trend toward the eradication of the infection emerges, ultimately ensuring sustained stability within the epidemiological landscape.

Uncertainty dynamics in energy planning models: An autoregressive and Markov chain modeling approach

This paper deals with the modeling of stochastic processes in long-term multi-stage energy planning problems when limited information is available about the distributions of such processes. Starting from simple estimates of variation intervals for uncertain parameters, such as energy demands and costs, we model the temporal correlation of these parameters through carefully constructed autoregressive (AR) models that respect the intervals defined in each period. We introduce a coefficient for “zigzag” effects in the evolution of uncertain processes, which controls the correlation across periods and also the likelihood of extreme scenarios. To preserve the convexity of the stochastic problem, we discretize the AR models associated with the cost parameters involved in the objective function by Markov chains. The resulting formulation is then solved using a Stochastic Dual Dynamic Programming (SDDP) algorithm available in the literature that handles finite-state Markov chains. Our numerical experiments, carried out on the Swiss energy system, show a very desirable adaptation strategy of investment decisions to uncertainty scenarios, a behavior that is not observed when the temporal correlation is ignored. Moreover, the solutions lead to better out-of-sample cost performances, especially on extreme scenario realizations, than the non-correlated ones which usually result in overcapacities to protect against high, but unlikely, parameter variations over time.

Heavy-traffic queue length behavior in a switch under Markovian arrivals

AbstractThis paper studies the input-queued switch operating under the MaxWeight algorithm when the arrivals are according to a Markovian process. We exactly characterize the heavy-traffic scaled mean sum queue length in the heavy-traffic limit, and show that it is within a factor of less than 2 from a universal lower bound. Moreover, we obtain lower and upper bounds that are applicable in all traffic regimes and become tight in the heavy-traffic regime.We obtain these results by generalizing the drift method recently developed for the case of independent and identically distributed arrivals to the case of Markovian arrivals. We illustrate this generalization by first obtaining the heavy-traffic mean queue length and its distribution in a single-server queue under Markovian arrivals and then applying it to the case of an input-queued switch.The key idea is to exploit the geometric mixing of finite-state Markov chains, and to work with a time horizon that is chosen so that the error due to mixing depends on the heavy-traffic parameter.

Truncated stochastically switching processes.

There are a large variety of hybrid stochastic systems that couple a continuous process with some form of stochastic switching mechanism. In many cases the system switches between different discrete internal states according to a finite-state Markov chain, and the continuous dynamics depends on the current internal state. The resulting hybrid stochastic differential equation(hSDE) could describe the evolution of a neuron's membrane potential, the concentration of proteins synthesized by a gene network, or the position of an active particle. Another major class of switching system is a search process with stochastic resetting, where the position of a diffusing or active particle is reset to a fixed position at a random sequence of times. In this case the system switches between a search phase and a reset phase, where the latter may be instantaneous. In this paper, we investigate how the behavior of a stochastically switching system is modified when the maximum number of switching (or reset) events in a given time interval is fixed. This is motivated by the idea that each time the system switches there is an additive energy cost. We first show that in the case of an hSDE, restricting the number of switching events is equivalent to truncating a Volterra series expansion of the particle propagator. Such a truncation significantly modifies the moments of the resulting renormalized propagator. We then investigate how restricting the number of reset events affects the diffusive search for an absorbing target. In particular, truncating a Volterra series expansion of the survival probability, we calculate the splitting probabilities and conditional MFPTs for the particle to be absorbed by the target or exceed a given number of resets, respectively.

Coverage performance of cooperative NOMA MmWave networks with wireless power transfer

The millimeter-wave (mmWave) plays a vital role in the fifth generation (5G) and beyond-5G (B5G) cellular networks, which is also attractive for wireless power transfer (WPT) since the mmWave base stations (BSs) are usually equipped with large antenna arrays. This paper studies a wireless-powered mmWave communication system with non-orthogonal multiple access (NOMA) cooperative transmission. In this system, the BS intends to transmit signals to a near-wireless device (WD) and a far-WD simultaneously, where the location of near-WD is fixed that can be treated as a relay for cooperatively transferring the signal to a far-WD, which is far away from the BS. Since there are multiple far-WDs in the region, two pairing schemes based on the distance between the near and far WDs are investigated, i.e., 1) near-WD pairs with the nearest far-WD (NNF); 2) near-WD pairs with a randomly selected far-WD (NRF). We evaluate the downlink transmission performance by assuming that the near-WD is energy-constrained and performs cooperative transmission after scavenging sufficient radio frequency (RF) energy while successfully decoding both the signals for near-WD and far-WD. Since the near-WD may accumulate sufficient energy in several transmission blocks, the processes of charging and discharging are analyzed by a finite-state Markov chain. On this basis, the performance of two WD pairing schemes is analyzed by deriving the exact expressions of coverage probabilities for both the near-WD and far-WD. The simulation results show that the considered mmWave cooperative communication system with NOMA schemes can achieve a better coverage performance than the compared OMA and other schemes, while the coverage probability of far-WD with the NNF pairing scheme is better than that with the NRF pairing scheme.

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.

Analytical Techniques for Performance Evaluation of Fountain-Coded File Distribution in Packet Radio Networks

Convenient analytical techniques are developed for use in preliminary design of systems and protocols for the delivery of fountain-coded files from a source node to a destination node with the assistance of a relay node. Each node has a half-duplex packet radio, and the time-varying disturbance on each link is represented by a lumpable finite-state Markov chain. The analytical techniques include the use of first passage times and recurrence times in the derivation of an exact expression for the completion time of file transfer over a radio link and the application of results on Markov-correlated Bernoulli trials to obtain exact expressions for completion times of relay-aided file transfer in wireless networks. In the development of analytical techniques, we employ capacity-achieving channel codes and ideal fountain codes; however, we present simulation results for packet radios that employ practical channel codes and for which the fountain code is a standard systematic raptor code.

Adaptive Modulation Based on Nondata-Aided Error Vector Magnitude for Smart Systems in Smart Cities

A smart city involves big data transmission (BDT) between smart systems, which increases queue delays and leads to difficulty in enhancing the spectral efficiency. Adaptive modulation is an effective technique for enhancing data transmission rates in smart systems. However, traditional adaptive modulation approaches are not suitable for BDT in smart systems because the delays caused by the large amount of transmitted data lead to difficulty in evaluating the channel quality. In this paper, we propose a nondata-aided error vector (NDA-EVM) that can be employed in adaptive modulation over wireless channels. The proposed NDA-EVM can be used to evaluate the channel quality and symbol error rate (SER), which reflect the quality of service (QoS) of the system. We formulated the relationship between the NDA-EVM and SER, which provides a basis for designing adaptive modulation techniques for smart systems. To address the low average spectrum efficiency (ASE) caused by BDT queue delays, an adaptive modulation strategy based on the finite-state Markov chain (FSMC) of the NDA-EVM (i.e., NDA-EVM-AM) was designed. This method simplifies the adaptive modulation algorithm for smart systems to search for the optimal transfer probability in the FSMC matrix based on two typical states: the resident state and transient state. Moreover, we proposed an analytical procedure to describe queuing behavior to analyze the performance of the NDA-EVM-AM algorithm for smart systems in smart cities. The performance is compared with that of a conventional adaptive modulation algorithm through simulations. The results show that compared with traditional adaptive modulation, NDA-EVM-AM obtains a lower packet loss rate and higher spectral efficiency for smart systems.

Three-dimensional stochastic Navier–Stokes equations with Markov switching

A finite-state Markov chain is introduced in the noise terms of the three-dimensional stochastic Navier–Stokes equations in order to allow for transitions between two types of multiplicative noises. We call such systems as stochastic Navier–Stokes equations with Markov switching. To solve such a system, a family of regularized stochastic systems is introduced. For each such regularized system, the existence of a unique strong solution (in the sense of stochastic analysis) is established by the method of martingale problems and pathwise uniqueness. The regularization is removed in the limit by obtaining a weakly convergent sequence from the family of regularized solutions, and identifying the limit as a solution of the three-dimensional stochastic Navier–Stokes equation with Markov switching.

A mixed singular/switching control problem with terminal cost for modulated diffusion processes

In this paper, we study the regularity of the value function associated with a stochastic control problem involving two controls that act simultaneously on a modulated multidimensional diffusion process. The first is a switching control modelling a random clock. Whenever the random clock rings, the generator matrix is replaced by another, leading to a different dynamic for the finite state Markov chain of the modulated diffusion process. The second is a singular stochastic control that is executed on the process within each regime.

Pricing European Vulnerable Options with Jumps and Stochastic Default Obstacles Barrier under Regime Switching

In this paper, we propose an enhanced model for pricing vulnerable options. Specifically, our model assumes that parameters such as interest rates, jump intensity, and asset value volatility are governed by an observable continuous-time finite-state Markov chain. We take into account European vulnerable options that are exposed to both default risk and rare shocks from underlying and counterparty assets. We also consider stochastic default barriers driven by a regime-switching model and geometric Brownian motion, thus improving upon the assumption of fixed default barriers. The risky assets follow a related jump-diffusion process, whereas the default barriers are influenced by a geometric Brownian motion correlated with the risky assets. Within the framework of our model, we derive an explicit pricing formula for European vulnerable options. Furthermore, we conduct numerical simulations to examine the effects of default barriers and other related parameters on option prices. Our findings indicate that stochastic default barriers increase credit risk, resulting in a decrease in option prices. By considering the aforementioned factors, our research contributes to a better understanding of pricing vulnerable options in the context of counterparty credit risk in over-the-counter trading.

Metastability from the large deviations point of view: A [formula omitted]-expansion of the level two large deviations rate functional of non-reversible finite-state Markov chains

Consider a sequence of continuous-time Markov chains (Xt(n):t≥0) evolving on a fixed finite state space V. Let In be the level two large deviations rate functional for Xt(n), as t→∞. Under a hypothesis on the jump rates, we prove that In can be written as In=I(0)+∑1≤p≤q(1/θn(p))I(p) for some rate functionals I(p). The weights θn(p) correspond to the time-scales at which the sequence of Markov chains Xt(n) exhibit a metastable behavior, and the zero level sets of the rate functionals I(p) identify the metastable states.

Joint Distribution of Ages of Information in Networks

We study a general setting of status updating systems in which a set of source nodes provide status updates about some physical process(es) to a set of monitors. The freshness of information available at each monitor is quantified in terms of the Age of Information (AoI), and the vector of AoI processes at the monitors (or equivalently the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">age vector</i> ) models the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">continuous</i> state of the system. While the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">marginal</i> distributional properties of each AoI process have been studied for a variety of settings using the stochastic hybrid system (SHS) approach, we lack a counterpart of this approach to systematically study their <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint</i> distributional properties. Developing such a framework is the main contribution of this paper. In particular, we model the discrete state of the system as a finite-state continuous-time Markov chain, and describe the coupled evolution of the continuous and discrete states of the system by a piecewise linear SHS with linear reset maps. Using the notion of tensors, we first derive first-order linear differential equations for the temporal evolution of both the joint moments and the joint moment generating function (MGF) for an arbitrary set of age processes. We then characterize the conditions under which the derived differential equations are asymptotically stable. The generality of our framework is demonstrated by recovering several existing results as special cases. Finally, we apply our framework to derive closed-form expressions of the stationary joint MGF in a multi-source updating system under non-preemptive and source-agnostic/ source-aware preemptive in service queueing disciplines.

A Markov chain approximation of switched Fokker–Planck equations for a model of on–off intermittency in the postural control during quiet standing

The intermittent on–off switching of feedback control is considered as a major mechanism of postural stabilization during human quiet standing, which can be modeled by switched-type hybrid stochastic delay differential equations with unstable subsystems. Dynamics of the model can also be described by the corresponding switched-type Fokker–Planck (FP) equations. Here, we develop a comprehensive numerical recipe to simulate switched-type FP equations in the case that the probability current is conserved at the switching boundary, as is the case for noise-free models exhibiting C0-continuity for solutions at the boundary, but in a way extendable to cases with discontinuous jumps. Specifically, the FP equations are approximated by a finite state Markov chain model using the finite element method. Then, dynamics of the Markov chain model, including time evolution of probability density function (PDF), stationary PDF, and power spectrum of postural sway are analyzed. We further investigate how the stationary PDF alters as values of important parameters of the model change. Dynamics of the Markov chain model are compared with Monte Carlo-based dynamics of the model, by which the developed numerical recipe is validated. The obtained Markov chain model forms a basis of our future investigations of the intermittent postural control as a Markov decision process.

Randomized Scheduling of Real-Time Traffic in Wireless Networks Over Fading Channels

Despite the rich literature on scheduling algorithms for wireless networks, algorithms that can provide deadline guarantees on packet delivery for general traffic and interference models are very limited. In this paper, we study the problem of scheduling real-time traffic under a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">conflict-graph interference model</i> with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unreliable links</i> due to channel fading. Packets that are not successfully delivered within their deadlines are of no value. We consider traffic (packet arrival and deadline) and fading (link reliability) processes that evolve as an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unknown</i> finite-state Markov chain. The performance metric is efficiency ratio which is the fraction of packets of each link which are delivered within their deadlines compared to that under the optimal (unknown) policy. We first show a conversion result that shows classical non-real-time scheduling algorithms can be ported to the real-time setting and yield a constant efficiency ratio. In particular, Max-Weight Scheduling () yields an efficiency ratio of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1/2$</tex-math> </inline-formula> . We then propose randomized algorithms that achieve efficiency ratios strictly higher than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1/2$</tex-math> </inline-formula> , by carefully randomizing over the maximal schedules. Further, we propose low-complexity and myopic distributed randomized algorithms, and characterize their efficiency ratio. Simulation results are presented that verify that the randomized algorithms outperform classical ones such as and for scheduling real-time traffic over fading channels.

Optimal power control for sensors and DoS attackers over a fading channel network

In this paper, we focus on the optimal power control for the sensors and attackers in a remote state estimation process, which guarantees the achievement of the estimation task under the denial-of-service (DoS) attack. In the remote estimation process under the DoS attack, each sensor observes the target state and transmits its local estimates to the remote estimator via a wireless network, communications in which are disrupted by jamming signals. For the wireless network, while most of the existing works on the resilient remote state estimation consider time-invariant channel state information (CSI) model, we pay attention to the fading channel network, in which the channel state is fixed and the channel model can be characterized by a generalized finite-state Markov chain. We model the conflicting nature among the sensor and the attacker as a general-sum stochastic game, proposing a Nash Q-learning algorithm to find the optimal power control strategies for the two players. Moreover, the monotone structure of these strategies is constructed with a sufficient condition. Taking into account the practical applications, we also consider the case where the sensor and the attacker only know partial information of each other and employ the Bayesian game to model it. Based on the channel information of each player and its believes of the channel distribution of the opponent, the type-contingent energy strategies are obtained. Finally, simulation results are given to verify our results.