Time-homogeneous Markov Process Research Articles

Feller evolution systems: Generators and approximation

A time and space inhomogeneous Markov process is a Feller evolution process, if the corresponding evolution system on the continuous functions vanishing at infinity is strongly continuous. We discuss generators of such systems and show that under mild conditions on the generators a Feller evolution can be approximated by Markov chains with Lévy increments.The result is based on the approximation of the time homogeneous spacetime process corresponding to a Feller evolution process. In particular, we show that a d-dimensional Feller evolution corresponds to a (d + 1)-dimensional Feller process. It is remarkable that, in general, this Feller process has a generator with discontinuous symbol.

A brief note on computing average state occupation times

AbstractOBJECTIVEThis review discusses how biometricians would probably compute or estimate expected waiting times, if they had the data.METHODSOur framework is a time-inhomogeneous Markov multistate model, where all transition hazards are allowed to be time-varying. We assume that the cumulative transition hazards are given. That is, they are either known, as in a simulation, determined by expert guesses, or obtained via some method of statistical estimation. Our basic tool is product integration, which transforms the transition hazards into the matrix of transition probabilities. Product integration enjoys a rich mathematical theory, which has successfully been used to study probabilistic and statistical aspects of multistate models. Our emphasis will be on practical implementation of product integration, which allows us to numerically approximate the transition probabilities. Average state occupation times and other quantities of interest may then be derived from the transition probabilities.(ProQuest: ... denotes formulae omitted.)1. IntroductionThis review is motivated by an interdisciplinary workshop on 'Multistate event history analysis' at the Netherlands Interdisciplinary Demographic Institute, The Hague, in April 2011. The workshop brought together demographers and biometricians, who both use multistate models, but appear to follow somewhat different methodological traditions.At the workshop, there were discussions on how to compute average state occupation times. The organizer, Frans Willekens, informed us that the most common approaches in demography use either piecewise linear approximations of the survival function or piece- wise constant transition hazards (personal communication). See also Gill and Keilman (1990) on these approaches, including a critique of the former method.We review how biometricians might compute expected waiting times, if they had the data. In biometry, the restriction is that right-censoring typically precludes evaluating waiting time distributions on the whole of their support. In other words, the maximum follow-up in most of the data sets analyzed by biometricians is considerably smaller than the assumed maximum age in the population under study. As a consequence, it is more common to consider median waiting times (e.g., Brookmeyer and Crowley 1982) or expectations restricted to a maximum follow-up (e.g., Andersen et al. 1993, Example IV.3.8). In the context of demography, expected (restricted) waiting times are arguably more relevant, for instance for policy making.The key idea is that a certain transformation, product integration, allows us to move from the transition hazards of a time-inhomogeneous Markov process to the matrix of transition probabilities. As also noted by Gill and Keilman (1990), combined with the initial distribution of the process, this allows us to derive expected waiting times in a given state and other quantities of interest.In demography, product integration appears to be rarely used. Gill and Keilman (1990) mention the relation, but then proceed to attack a different problem, namely estimation of constant transition hazards with population registry data. In a recent tutorial on multistate methods, Kuo, Suchindran, and Koo (2008) value product integration as a 'basic tool', but argue that it 'is difficult to implement.' These authors therefore proceed to work under special assumptions such as uniform right-censoring. Another reference is Schoen (2005) who mentions product integration as a tool for numerical evaluation, but then concentrates on special cases with an analytical solution.2. The relation between transition hazards and transition probabilitiesConsider a time-inhomogeneous Markov process (Xt)t≥0 with state space {0, 1,2, . . . ,J}. We assume that (Xt )t≥0 has right-continuous sample paths, which are constant between transition times. That is, if the process moves from state j to state k, j = k, at time t0 , Xt0 = k and Xt0 - = j . …

Application of Markov Process in Performance Analysis of Feeding System of Sugar Industry

To analyse the performance measures of complex repairable systems having more than two states, that is, working, reduced and failed, it is essential to model suitably their states so that the system governs a stochastic process. In this paper, the application of time-homogeneous Markov process is used to express reliability and availability of feeding system of sugar industry involving reduced states and it is found to be a powerful method that is totally based on modelling and numerical analysis. The selection of appropriate units/components in designing a system with different characteristics is necessary for the system analyst to maintain the failure-free operation. Keeping this concept in this study, the steady state availability of concern system is analysed and optimized by using a popular search technique, genetic algorithm. The objective of this paper is to consider the system operative process as Markov process and find its reliability function and steady state availability in a very effective manner and also to obtain an optimal system designing constituents which will allow a failure-free operation for long time period as required for maximum system productivity. The system performance measures and optimized design parameters are described and obtained here by considering an illustrative example.

Achievable Throughput of Energy Harvesting Cognitive Radio Networks

We consider energy harvesting cognitive radio networks to improve both energy efficiency and spectral efficiency. The goal of this paper is to analyze the theoretically achievable throughput of the secondary transmitter, which harvests energy from ambient sources or wireless power transfer systems while opportunistically accessing the spectrum licensed to the primary network. By modeling the temporal correlation of the primary traffic according to a time-homogeneous discrete Markov process, we derive the upper bound on the achievable throughput as a function of the energy arrival rate, the temporal correlation of the primary traffic, and the detection threshold for a spectrum sensor. The optimal detection threshold is then derived to maximize the upper bound on the achievable throughput under an energy causality constraint and a collision constraint. The energy causality constraint mandates that the total consumed energy should not exceed the total harvested energy, while the collision constraint is required to protect the primary network from secondary transmission. Analytical results show the temporal correlation of the primary traffic to enable efficient usage of the harvested energy by preventing the secondary transmitter from accessing the spectrum that may be occupied by the primary network.

STOCHASTIC LOCAL INTENSITY LOSS MODELS WITH INTERACTING PARTICLE SYSTEMS

It is well known from the work of Schönbucher that the marginal laws of a loss process can be matched by a unit increasing time inhomogeneous Markov process, whose deterministic jump intensity is called local intensity. The stochastic local intensity (SLI) models such as the one proposed by Arnsdorf and Halperin allow to get a stochastic jump intensity while keeping the same marginal laws. These models involve a nonlinear stochastic differential equation (SDE) with jumps. The first contribution of this paper is to prove the existence and uniqueness of such processes. This is made by means of an interacting particle system, whose convergence rate toward the nonlinear SDE is analyzed. Second, this approach provides a powerful way to compute pathwise expectations with the SLI model: we show that the computational cost is roughly the same as a crude Monte Carlo algorithm for standard SDEs.

Optimal Spectrum Access for Energy Harvesting Cognitive Radio Networks

We consider energy harvesting cognitive radio networks in which a secondary transmitter harvests energy from ambient sources or wireless power transfer systems while opportunistically accessing the spectrum licensed to the primary network. The primary traffic is modeled as a time-homogeneous discrete Markov process, and the secondary transmitter may not be able to operate continuously due to sporadic and unstable energy sources. At the beginning of each time slot, the secondary transmitter thus needs to determine whether to remain idle so as to conserve energy, or to execute spectrum sensing to acquire knowledge of the current spectrum occupancy state. It also needs to configure the spectrum sensor detection threshold to achieve an effective tradeoff between false alarms and misdetections. This sequential decision-making, done to maximize the expected total throughput, requires the joint design of a spectrum sensing policy and a detection threshold under the energy causality and collision constraints. We formulate this stochastic optimization problem as a constrained partially observable Markov decision process (POMDP), and then convert it to a computationally tractable unconstrained POMDP. Numerical results show that the proposed approach enables efficient usage of the harvested energy by exploiting the temporal correlation of the primary traffic.

Unscented Kalman Filter Over Unreliable Communication Networks With Markovian Packet Dropouts

The paper investigates the stability of the discrete-time modified unscented Kalman filter (MUKF) in transmitting measurement outputs to the filter via a network. Since the communication channel states do not always vary in time, the arrival of the observations is modeled as a two state time-homogeneous Markov process γk. The stability of the estimation error covariance matrices at packet reception times is analyzed. Sufficient conditions (related to Hk and the initial conditions) for the peak covariance stability and the usual covariance stability are given. Also the relationship between the different types of stability notions is illustrated for systems with i.i.d observation dropouts. Numerical example is given to illustrate the effectiveness of the techniques developed.

Consensus condition for linear multi-agent systems over randomly switching topologies

This paper studies both continuous and discrete time consensus problems for multi-agent systems with linear time-invariant agent dynamics over randomly switching topologies. The switching is governed by a time-homogeneous Markov process, whose state corresponds to a possible interaction topology among agents. Necessary and sufficient conditions are derived for achieving consensus under a common control protocol, respectively. It is shown that the effect of switching topologies on consensus is determined by the union of topologies associated with the positive recurrent states of the Markov process. Moreover, the effect of random link failures on discrete time consensus is investigated. The implications and relationships with the existing results are discussed. Finally, the theoretical results are validated via simulations.

The embedding problem for markov models of nucleotide substitution.

Continuous-time Markov processes are often used to model the complex natural phenomenon of sequence evolution. To make the process of sequence evolution tractable, simplifying assumptions are often made about the sequence properties and the underlying process. The validity of one such assumption, time-homogeneity, has never been explored. Violations of this assumption can be found by identifying non-embeddability. A process is non-embeddable if it can not be embedded in a continuous time-homogeneous Markov process. In this study, non-embeddability was demonstrated to exist when modelling sequence evolution with Markov models. Evidence of non-embeddability was found primarily at the third codon position, possibly resulting from changes in mutation rate over time. Outgroup edges and those with a deeper time depth were found to have an increased probability of the underlying process being non-embeddable. Overall, low levels of non-embeddability were detected when examining individual edges of triads across a diverse set of alignments. Subsequent phylogenetic reconstruction analyses demonstrated that non-embeddability could impact on the correct prediction of phylogenies, but at extremely low levels. Despite the existence of non-embeddability, there is minimal evidence of violations of the local time homogeneity assumption and consequently the impact is likely to be minor.

Ornstein–Uhlenbeck processes time changed with additive subordinators and their applications in commodity derivative models

We characterize Ornstein–Uhlenbeck processes time changed with additive subordinators as time-inhomogeneous Markov semimartingales, based on which a new class of commodity derivative models is developed. Our models are tractable for pricing European, Bermudan and American futures options. Calibration examples show that they can be better alternatives than those developed in Li and Linetsky (2012) [6]. Our method can be applied to many other processes popular in various areas besides finance to develop time-inhomogeneous Markov processes with desirable features and tractability.

Quantized Feedback Control Over Packet Dropout Communication Channels

This paper investigates quantized feedback control problems for linear time-invariant control systems, where the sensors and controllers are geographically separated and connected via noisy, bandwidth-limited digital communication channels. The packet dropout process of the channel is modeled as a time-homogeneous Markov process. An adaptive differential coding strategy and a predictive control policy are implemented to achieve the minimum data rate of the channel for mean square stabilization of the unstable plant. In particular, it is shown that a sufficient condition on mean square stabilization of the system with disturbances is that the data rate is more than the lower bound given in our results. The sufficient condition decomposes into two terms: a condition on the data rate and a condition on the transition probabilities of the Markov chain. An illustrative example is given to demonstrate the effectiveness of the proposed method. DOI: http://dx.doi.org/10.5755/j01.itc.41.3.1064

Stochastic modeling of Pseudomonas syringae growth in the phyllosphere

Pseudomonas syringae is a gram-negative bacterium which lives on leaf surfaces. Its growth has been described using epifluorescence microscopy and image analysis; it was found to be growing in aggregates of a wide range of sizes. We develop a stochastic model to describe aggregate distribution and determine the mechanisms generating experimental observations. We found that a logistic birth–death model with migration (time-homogeneous Markov process) provides the best description of the observed data. We discuss how to analyze the joint distribution of the numbers of aggregates of different sizes at a given time and explore how to account for new aggregates being created, that is, the joint distribution of the family size statistics conditional on the total number of aggregates. We compute the first two moments. Through simulations we examine how the model’s parameters affect the aggregate size distribution and successfully explain the quantitative experimental data available. Aggregation formation is thought to be the first step towards pathogenic behavior of this bacterium; understanding aggregate size distribution would prove useful to understand the switch from epiphytic to pathogenic behavior.

HEAT KERNEL INTEREST RATE MODELS WITH TIME-INHOMOGENEOUS MARKOV PROCESSES

We consider a heat kernel approach for the development of stochastic pricing kernels. The kernels are constructed by positive propagators, which are driven by time-inhomogeneous Markov processes. We multiply such a propagator with a positive, time-dependent and decreasing weight function, and integrate the product over time. The result is a so-called weighted heat kernel that by construction is a supermartingale with respect to the filtration generated by the time-inhomogeneous Markov processes. As an application, we show how this framework naturally fits the information-based asset pricing framework where time-inhomogeneous Markov processes are utilized to model partial information about random economic factors. We present examples of pricing kernel models which lead to analytical formulae for bond prices along with explicit expressions for the associated interest rate and market price of risk. Furthermore, we also address the pricing of fixed-income derivatives within this framework.

Research on stochastic models and algorithms for web page ranking

As the World Wide Web grows rapidly, search engines have become the most popular tools to access the large volume of information from it. And the key factor of the search engine is the ranking model of web pages, which contains two types: static rank and dynamic rank. In past, different approaches have been designed to solve these two problems separately. In this thesis, we analyze them on the same base—stochastic process model, and design new algorithms to solve them efficiently and effectively. Firstly, we establish a framework on Markov skeleton process to compute the page importance by investigating real browsing behaviors of users. Within this framework, we design a group of eight new novel algorithms all referred to as BrowseRank, to compute the page importance based on the continuous-time time-homogeneous Markov process, which is one of three special cases of the Markov skeleton process. And from the experimental results, we find BrowseRank outperforms other baseline algorithms, such as PageRank and TrustRank. Secondly, we build a supervised framework for rank aggregation based on Markov chain. Within this framework, we not only generalize some unsupervised algorithms to supervised ones, but also design a new approach referred to as Supervised MC2 for rank aggregation, which transform the original NP-hard problem to a semi-defined programme.

Markov processes and generalized Schrödinger equations

Starting from the forward and backward infinitesimal generators of bilateral, time-homogeneous Markov processes, the self-adjoint Hamiltonians of the generalized Schrödinger equations are first introduced by means of suitable Doob transformations. Then, by broadening with the aid of the Dirichlet forms, the results of the Nelson stochastic mechanics, we prove that it is possible to associate bilateral, and time-homogeneous Markov processes to the wave functions stationary solutions of our generalized Schrödinger equations. Particular attention is then paid to the special case of the Lévy-Schrödinger (LS) equations and to their associated Lévy-type Markov processes, and to a few examples of Cauchy background noise.

Mean square stability for Kalman filtering with Markovian packet losses

This paper studies the stability of Kalman filtering over a network subject to random packet losses, which are modeled by a time-homogeneous ergodic Markov process. For second-order systems, necessary and sufficient conditions for stability of the mean estimation error covariance matrices are derived by taking into account the system structure. While for certain classes of higher-order systems, necessary and sufficient conditions are also provided to ensure stability of the mean estimation error covariance matrices. All stability criteria are expressed by simple inequalities in terms of the largest eigenvalue of the open loop matrix and transition probabilities of the Markov process. Their implications and relationships with related results in the literature are discussed.

Stationary response of strongly non-linear oscillator with fractional derivative damping under bounded noise excitation

The stochastic averaging method for strongly non-linear oscillators with lightly fractional derivative damping of order α (0<α≤1) subject to bounded noise excitations is proposed by using the generalized harmonic function. The system state is approximated by a two-dimensional time-homogeneous diffusion Markov process of amplitude and phase difference using the proposed stochastic averaging method. The approximate stationary probability density of response is obtained by solving the reduced Fokker–Planck–Kolmogorov (FPK) equation using the finite difference method and successive over relaxation method. A Duffing oscillator is taken as an example to show the application and validity of the method. In the case of primary resonance, the stochastic jump of the Duffing oscillator with fractional derivative damping and its P-bifurcation as the system parameters change are examined for the first time using the stationary probability density of amplitude.

Stochastic jump and bifurcation of Duffing oscillator with fractional derivative damping under combined harmonic and white noise excitations

The stochastic jump and bifurcation of Duffing oscillator with fractional derivative damping of order α (0< α <1) under combined harmonic and white noise excitations are studied. First, the system state is approximately represented by two-dimensional time-homogeneous diffusive Markov process of amplitude and phase difference using the stochastic averaging method. Then, the method of reduced Fokker–Plank–Kolmogorov (FPK) equation is used to predict the stationary response of the original system. The phenomenon of stochastic jump and bifurcation as the fractional orders' change is examined. ► The stochastic jump and bifurcation as the fractional order change are observed. ► The stochastic jump and bifurcation are explained using the stationary probability density. ► The stochastic jump and bifurcation are examined using the samples of displacement.

On a Unique Ergodicity of Some Markov Processes

It is proved that the sufficient condition for the uniqueness of an invariant measure for Markov processes with the strong asymptotic Feller property formulated by Hairer and Mattingly (Ann Math 164(3):993–1032, 2006) entails the existence of at most one invariant measure for e-processes as well. Some application to time-homogeneous Markov processes associated with a nonlinear heat equation driven by an impulsive noise is also given.

On the First Exit Time of a Nonnegative Markov Process Started at a Quasistationary Distribution

Let {M n } n≥0 be a nonnegative time-homogeneous Markov process. The quasistationary distributions referred to in this note are of the form Q A (x) = lim n→∞P(M n ≤ x | M 0 ≤ A, M 1 ≤ A, …, M n ≤ A). Suppose that M 0 has distribution Q A , and define T A Q A = min{n | M n &amp;gt; A, n ≥ 1}, the first time when M n exceeds A. We provide sufficient conditions for Q A (x) to be nonincreasing in A (for fixed x) and for T A Q A to be stochastically nondecreasing in A.