Continuous-time Finite-state Markov Research Articles

Joint Distribution of Ages of Information in Networks

We study a general setting of status updating systems in which a set of source nodes provide status updates about some physical process(es) to a set of monitors. The freshness of information available at each monitor is quantified in terms of the Age of Information (AoI), and the vector of AoI processes at the monitors (or equivalently the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">age vector</i> ) models the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">continuous</i> state of the system. While the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">marginal</i> distributional properties of each AoI process have been studied for a variety of settings using the stochastic hybrid system (SHS) approach, we lack a counterpart of this approach to systematically study their <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint</i> distributional properties. Developing such a framework is the main contribution of this paper. In particular, we model the discrete state of the system as a finite-state continuous-time Markov chain, and describe the coupled evolution of the continuous and discrete states of the system by a piecewise linear SHS with linear reset maps. Using the notion of tensors, we first derive first-order linear differential equations for the temporal evolution of both the joint moments and the joint moment generating function (MGF) for an arbitrary set of age processes. We then characterize the conditions under which the derived differential equations are asymptotically stable. The generality of our framework is demonstrated by recovering several existing results as special cases. Finally, we apply our framework to derive closed-form expressions of the stationary joint MGF in a multi-source updating system under non-preemptive and source-agnostic/ source-aware preemptive in service queueing disciplines.

Managing resource allocation for the recruitment stocking problem

This paper revisits the Recruitment Stocking Problem (RSP), in which a prescribed total number of kits must be allocated across multiple distribution sites, where streams of subjects arrive for kit pickup. A kit can be a test-kit for a new pharmaceutical drug, a medical device, a vaccine, a humanitarian relief survival kit, etc. RSP seeks to minimize the expected recruitment time — the time to recruit a prescribed target number of kits — by identifying an optimal or near-optimal kit allocation across all locations. To this end, we first develop a computational method for evaluating the expected recruitment time using the so-called Randomization Procedure. Specifically, since the recruitment process forms a finite-state continuous-time Markov process, the Randomization Procedure transforms the Markov process with varied transition rates from different states into one with a constant transition rate across all states, and thus substantially simplifies the RSP statistics computation. Further, we develop two search heuristics to efficiently search for optimal or near-optimal kit allocations, especially for large RSP models that are too computationally expensive to optimize by exhaustive search. Finally, we illustrate the efficacy of the proposed heuristics by numerical experiments, and then discuss practical implications and provide managerial insights.

Hitting times, number of jumps, and occupation times for continuous-time finite state Markov chains

In this paper we discuss three random variables associated with a finite state continuous-time Markov chain having a constant transition rate matrix: the number of jumps between two different states, the amount of time spent in a given state, known as an occupation time, and the time it takes for the Markov chain to first reach a particular state, known as a hitting time or first passage time. First, we calculate expected values for each of these random variables. We then derive the moment generating functions for these variables. Our approach is to use the vector/matrix representations of the various processes. Each solution is written in terms of an integral of the exponential of a matrix.

Optimal Investment and Proportional Reinsurance in a Regime-Switching Market Model under Forward Preferences

In this paper, we study the optimal investment and reinsurance problem of an insurance company whose investment preferences are described via a forward dynamic exponential utility in a regime-switching market model. Financial and actuarial frameworks are dependent since stock prices and insurance claims vary according to a common factor given by a continuous time finite state Markov chain. We construct the value function and we prove that it is a forward dynamic utility. Then, we characterize the optimal investment strategy and the optimal proportional level of reinsurance. We also perform numerical experiments and provide sensitivity analyses with respect to some model parameters.

Mean-Variance Asset-Liability Management in a Non-Markovian Regime-Switching Jump-Diffusion Market with Random Horizon

This paper investigates the problem of mean-variance asset-liability management (ALM) in a market where the dynamics of assets are non-Markovian regime-switching models driven by a Brownian motion, a Poisson random measure and a continuous time finite-state Markov chain. It is assumed that the time horizon is uncertain relying not only on asset prices and liability values, but also on other factors. The insurer aims to minimize the variance of the terminal surplus given an expected terminal surplus subject to the risk of paying out random liabilities of an extensive Cramer-Lundberg model. By further developing the solvability of a linear backward stochastic differential equation (BSDE) with two types of jumps, namely jumps modelled by a Poisson random measure and by basic martingales related to a Markov chain, closed-form expressions for both the efficient strategy and the efficient frontier are obtained and represented by unique solutions to several relevant BSDEs.

Optimal reduction of public debt under partial observation of the economic growth

We consider a government that aims at reducing the debt-to-(gross domestic product) (GDP) ratio of a country. The government observes the level of the debt-to-GDP ratio and an indicator of the state of the economy, but does not directly observe the development of the underlying macroeconomic conditions. The government’s criterion is to minimise the sum of the total expected costs of holding debt and of debt reduction policies. We model this problem as a singular stochastic control problem under partial observation. The contribution of the paper is twofold. Firstly, we provide a general formulation of the model in which the level of the debt-to-GDP ratio and the value of the macroeconomic indicator evolve as a diffusion and a jump-diffusion, respectively, with coefficients depending on the regimes of the economy. The latter are described through a finite-state continuous-time Markov chain. We reduce the original problem via filtering techniques to an equivalent one with full information (the so-called separated problem), and we provide a general verification result in terms of a related optimal stopping problem under full information. Secondly, we specialise to a case study in which the economy faces only two regimes and the macroeconomic indicator has a suitable diffusive dynamics. In this setting, we provide the optimal debt reduction policy. This is given in terms of the continuous free boundary arising in an auxiliary fully two-dimensional optimal stopping problem.

Near-optimal control problems for forward-backward regime-switching systems

This paper investigates the near-optimality for a class of forward-backward stochastic differential equations (FBSDEs) with continuous-time finite state Markov chains. The control domains are not necessarily convex and the control variables do not enter forward diffusion term. Some new estimates for state and adjoint processes arise naturally when we consider the near-optimal control problem in the framework of regime-switching. Inspired by Ekeland’s variational principle and a spike variational technique, the necessary conditions are derived, which imply the near-minimum condition of the Hamiltonian function in an integral sense. Meanwhile, some certain convexity conditions and the near-minimum condition are sufficient for the near-optimal controls with order ε1/2. A recursive utility investment consumption problem is discussed to illustrate the effectiveness of our theoretical results.

Equilibrium investment and risk control for an insurer with non-Markovian regime-switching and no-shorting constraints

In this paper, we study the time-consistent equilibrium investment and risk control policies for an insurer under the mean-variance criterion. The dynamics of liability and assets are given by non-Markovian regime-switching models driven by a Brownian motion and a continuous time finite-state Markov chain. It is assumed that the insurer has a time-varying preference of risk depending dynamically on current wealth, and is not allowed to short sell the risky assets. With the aid of backward stochastic differential equations and bounded mean oscillation martingales, we obtain feedback representations of the open-loop equilibrium strategies. Finally, the result is also applied to solve an example of the Markovian regime-switching model.

The CTMC-Heston Model: Calibration and Exotic Option Pricing with SWIFT

This work presents an efficient computational framework for pricing a general class of exotic and vanilla options under a versatile stochastic volatility model. In particular, we propose the use of a finite state continuous time Markov chain (CTMC) model, which closely approximates the classic Heston model but enables a simplified approach for consistently pricing a wide variety of financial derivatives. Under the CTMC–Heston model, we show that the shape of the implied volatility is preserved (hence, it has an equivalent ability to calibrate market smiles), yet it may price complex derivatives such as Asian options, variance swaps/options and cliquets with great efficiency. To accomplish this, we employ Markov chain approximation techniques, which have been gaining traction in the recent literature. We expect that this framework will have significant practical appeal, given the widespread adoption of the Heston model and the present difficulties that arise in pricing complex products.

Dynamical behavior of a hybrid switching SIS epidemic model with vaccination and Lévy jumps

In this paper, the dynamical behavior of a hybrid switching SIS epidemic model with vaccination and Lévy jumps is considered. Besides a standard geometric Brownian motion, another two driving processes are taken into account: a stationary Poisson point process and a continuous time finite-state Markov chain. Firstly, we establish sufficient conditions for persistence in the mean of the disease. Then we obtain sufficient conditions for extinction of the disease. In addition, we also establish sufficient conditions for the existence of positive recurrence of the solutions to the model by constructing a suitable stochastic Lyapunov function with regime switching.

Pairs Trading Under Drift Uncertainty and Risk Penalization

In this work, we study the dynamic portfolio optimization problem related to the pairs trading, which is an investment strategy that matches a long position in one security with a short position in an another security with similar characteristics. The relation between pairs, called spread, is modeled by a Gaussian mean-reverting process whose drift rate is modulated by an unobservable continuous-time finite state Markov chain. Using the classical stochastic filtering theory, we reduce this problem with partial information to the one with complete information and solve it for the logarithmic utility function, where the terminal wealth is penalized by the riskiness of the portfolio according to the realized volatility of the wealth process. We characterize optimal dollar-neutral strategies as well as optimal value functions under both full and partial information and show that the certainty principle holds for the optimal portfolio strategy. Finally, we provide a numerical analysis for a simple example with a two-state Markov chain.

Optimal switching under a hybrid diffusion model and applications to stock trading

This paper is concerned with the optimal switching problem under a hybrid diffusion (or, regime switching) model in an infinite horizon. The state of the system consists of a number of diffusions coupled by a finite-state continuous-time Markov chain. Based on the dynamic programming principle, the value function of our optimal switching problem is proved to be the unique viscosity solution to the associated system of variational inequalities. The optimal switching strategy, indicating when and where it is optimal to switch, is given in terms of the switching and continuation regions. In many applications, the underlying Markov chain has a large state space and exhibits two-time-scale structure. In this case, a singular perturbation approach is employed to reduce the computational complexity involved. It is shown that as the time-scale parameter ε goes to zero, the value function of the original problem converges to that of a limit problem. The limit problem is much easier to solve, and its optimal switching solution leads to an approximate solution to the original problem. Finally, as an application of our theoretical results, an example concerning the stock trading problem in a regime switching market is provided. It is emphasized that, this paper is the first time to introduce the optimal switching as a general framework to study the stock trading problem, in view of their inherent connection. Both optimal trading rule and convergence result are numerically demonstrated in this example.

A note on chaotic and predictable representations for Itô–Markov additive processes

ABSTRACTIn this article, we provide predictable and chaotic representations for Itô–Markov additive processes X. Such a process is governed by a finite-state continuous time Markov chain J which allows one to modify the parameters of the Itô-jump process (in so-called regime switching manner). In addition, the transition of J triggers the jump of X distributed depending on the states of J just prior to the transition. This family of processes includes Markov modulated Itô–Lévy processes and Markov additive processes. The derived chaotic representation of a square-integrable random variable is given as a sum of stochastic integrals with respect to some explicitly constructed orthogonal martingales. We identify the predictable representation of a square-integrable martingale as a sum of stochastic integrals of predictable processes with respect to Brownian motion and power-jumps martingales related to all the jumps appearing in the model. This result generalizes the seminal result of Jacod–Yor and is of importance in financial mathematics. The derived representation then allows one to enlarge the incomplete market by a series of power-jump assets and to price all market-derivatives.

A system of nonlinear equations with application to large deviations for Markov chains with finite lifetime

In this paper, we first show the existence of solutions to the following system of nonlinear equations{a11x1+a12x2+⋯+a1nxn=b111x1+b121x2+⋯+b1n1xn,a211x1+a22x2x1+⋯+a2nxnx1=b21x1+b22x1x2+⋯+b2nx1xn,⋯⋯ak,k−11xk−1+∑1≤j≤nj≠k−1akjxjxk−1=bk,k−1xk−1+∑1≤j≤nj≠k−1bkjxk−1xj,⋯⋯an,n−11xn−1+∑1≤j≤nj≠n−1anjxjxn−1=bn,n−1xn−1+∑1≤j≤nj≠n−1bnjxn−1xj, where n≥2 and aij,bij,1≤i,j≤n, are positive constants. Then, we make use of this result to obtain the large deviation principle for the occupation time distributions of continuous-time finite state Markov chains with finite lifetime.

Measure distorted arrival rate risks and their rewards

Risks embedded in asset price dynamics are taken to be accumulations of surprise jumps. A Markov pure jump model is formulated on making variance gamma parameters deterministic functions of the price level. Estimation is done by matrix exponentiation of the transition rate matrix for a continuous time finite state Markov chain approximation. The motion is decomposed into a space dependent drift and a space dependent martingale component. Though there is some local mean reversion in the drift, space dependence of the martingale component renders the dynamics to be of the momentum type. Local risk is measured using market calibrated measure distortions that introduce risk charges into the lower and upper prices of two price economies. These risks are compensated by the exponential variation of space dependent arrival rates. Estimations are conducted for the S&P 500 index (SPX), the exchange traded fund for the financial sector (XLF), J. P. Morgan stock prices (JPM), the ratio of JPM to XLF, and the ratio of XLF to SPX.

How old is this bird? The age distribution under some phase sampling schemes

In this paper, we use a finite-state continuous-time Markov chain with one absorbing state to model an individual's lifetime. Under this model, the time of death follows a phase-type distribution, and the transient states of the Markov chain are known as phases. We then attempt to provide an answer to the simple question "What is the conditional age distribution of the individual, given its current phase"? We show that the answer depends on how we interpret the question, and in particular, on the phase observation scheme under consideration. We then apply our results to the computation of the age pyramid for the endangered Chatham Island black robin Petroica traversi during the monitoring period 2007-2014.

Markov fluid queue model of an energy harvesting IoT device with adaptive sensing

Energy management is key in prolonging the lifetime of an energy harvesting Internet of Things (IoT) device with rechargeable batteries. Such an IoT device is required to fulfill its main functionalities, i.e., information sensing and dissemination at an acceptable rate, while keeping the probability that the node first becomes non-operational, i.e., the battery level hits zero the first time within a given finite time horizon, below a desired level. Assuming a finite-state Continuous-Time Markov Chain (CTMC) model for the Energy Harvesting Process (EHP), we propose a risk-theoretic Markov fluid queue model for the computation of first battery outage probabilities in a given finite time horizon. The proposed model enables the performance evaluation of a wide spectrum of energy management policies including those with sensing rates depending on the instantaneous battery level and/or the state of the energy harvesting process. Moreover, an engineering methodology is proposed by which optimal threshold-based adaptive sensing policies are obtained that maximize the information sensing rate of the IoT device while meeting a Quality of Service (QoS) constraint given in terms of first battery outage probabilities. Numerical results are presented for the validation of the analytical model and also the proposed engineering methodology, using a two-state CTMC-based EHP.

Arrival Rate Functions

Asset price dynamics are taken to be accumulations of surprise jumps in the logarithm of prices. A Markov pure jump model is formulated on making variance gamma parameters deterministic functions of the price level. Estimation is done by matrix exponentiation of the transition rate matrix for a continuous time finite state Markov chain approximation. The motion is decomposed into a space dependent drift and a space dependent martingale component. Though there is some local mean reversion by and large the dynamics estimated is that of the momentum type. Risk compensation is seen by a linear relation between the exponential variation and measure distorted variations for the bid and ask prices of two price economies. Estimations are conducted for the S&P 500 index (SPX), the exchange traded fund for the financial sector (XLF), J. P. Morgan stock prices (JPM), the ratio of JPM to XLF and the ratio of XLF to SPX.

Local risk minimization for vulnerable European contingent claims on nontradable assets under regime switching models

ABSTRACTThis article focuses on an optimal hedging problem of the vulnerable European contingent claims. The underlying asset of the vulnerable European contingent claims is assumed to be nontradable. The interest rate, the appreciation rate and the volatility of risky assets are modulated by a finite-state continuous-time Markov chain. By using the local risk minimization method, we obtain an explicit closed-form solution for the optimal hedging strategies of the vulnerable European contingent claims. Further, we consider a problem of hedging for a vulnerable European call option. Optimal hedging strategies are obtained. Finally, a numerical example for the optimal hedging strategies of the vulnerable European call option in a two-regime case is provided to illustrate the sensitivities of the hedging strategies.

A Study on Estimating the Next Failure Time of Compressor Equipment in an Offshore Plant

The offshore plant equipment usually has a long life cycle. During its O&amp;M (Operation and Maintenance) phase, since the accidental occurrence of offshore plant equipment causes catastrophic damage, it is necessary to make more efforts for managing critical offshore equipment. Nowadays, due to the emerging ICTs (Information Communication Technologies), it is possible to send health monitoring information to administrator of an offshore plant, which leads to much concern on CBM (Condition-Based Maintenance). This study introduces three approaches for predicting the next failure time of offshore plant equipment (gas compressor) with case studies, which are based on finite state continuous time Markov model, linear regression method, and their hybrid model.