Independent Markov Chains Research Articles

Random walks colliding before getting trapped

Let $P$ be the transition matrix of a finite, irreducible and reversible Markov chain. We say the continuous time Markov chain $X$ has transition matrix $P$ and speed $\lambda $ if it jumps at rate $\lambda $ according to the matrix $P$. Fix $\lambda _X,\lambda _Y,\lambda _Z\geq 0$, then let $X,Y$ and $Z$ be independent Markov chains with transition matrix $P$ and speeds $\lambda _X,\lambda _Y$ and $\lambda _Z$ respectively, all started from the stationary distribution. What is the chance that $X$ and $Y$ meet before either of them collides with $Z$? For each choice of $\lambda _X,\lambda _Y$ and $\lambda _Z$ with $\max (\lambda _X,\lambda _Y)>0$, we prove a lower bound for this probability which is uniform over all transitive, irreducible and reversible chains. In the case that $\lambda _X=\lambda _Y=1$ and $\lambda _Z=0$ we prove a strengthening of our main theorem using a martingale argument. We provide an example showing the transitivity assumption cannot be removed for general $\lambda _X,\lambda _Y$ and $\lambda _Z$.

Consensus in second‐order Markovian jump multi‐agent systems via impulsive control using sampled information with heterogenous delays

AbstractThis paper investigates the consensus problem of second‐order Markovian jump multi‐agent systems with delays. Both network‐induced random delay and node‐induced state delay are considered, where the network‐induced random delay, subjected to a Markov chain, exists in the switching signal and the node‐induced state delay, related to switching topologies, is heterogeneous between any two linked agents. In order to reduce communication and control energy, an impulsive protocol is proposed, where each agent only can get delayed relative positions to neighbors and the velocity of itself at impulsive instants. By performing three steps of model transformation and introducing a mapping for two independent Markov chains, the consensus problem of the original continuous‐time system is equivalent to the stability problem of a discrete‐time expand error system with two Markovian jumping parameters and a necessary and sufficient criterion is derived. A numerical example is given to illustrate the effectiveness of the theoretical result.@@@@This work is supported by the National Natural Science Foundation of China under Grants 61374171, 61572210, and 51537003, the Fundamental Research Funds for the Central Universities (2015TS030), and the Program for Changjiang Scholars and Innovative Research Team in University (IRT1245).

Optimal stopping of switching diffusions with state dependent switching rates

This paper is concerned with a continuous-time and infinite-horizon optimal stopping problem in switching diffusion models. In contrast to the assumption commonly made in the literature that the regime-switching is modeled by an independent Markov chain, we consider in this paper the case of state-dependent regime-switching. The Hamilton–Jacobi–Bellman (HJB) equation associated with the optimal stopping problem is given by a system of coupled variational inequalities. By means of the dynamic programming (DP) principle, we prove that the value function is the unique viscosity solution of the HJB system. As an interesting application in mathematical finance, we examine the problem of pricing perpetual American put options with state-dependent regime-switching. A numerical procedure is developed based on the DP approach and an efficient discrete tree approximation of the continuous asset price process. Numerical results are reported.

The Value of Markov Chain Games with Incomplete Information on Both Sides

We consider zero-sum repeated games with incomplete information on both sides, where the states privately observed by each player follow independent Markov chains. It generalizes the model, introduced by Aumann and Maschler in the sixties and solved by Mertens and Zamir in the seventies, where the private states of the players were fixed. It also includes the model introduced in Renault [19] [Renault J (2006) The value of Markov chain games with lack of information on one side. Math. Oper. Res. 31(3):490–512.], of Markov chain repeated games with lack of information on one side, where only one player privately observes the sequence of states. We prove here that the limit value exists, and we obtain a characterization via the Mertens-Zamir system, where the “nonrevealing value function” plugged in the system is now defined as the limit value of an auxiliary “nonrevealing” dynamic game. This nonrevealing game is defined by restricting the players not to reveal any information on the limit behavior of their own Markov chain, as in Renault [19]. There are two key technical difficulties in the proof: (1) proving regularity, in the sense of equicontinuity, of the T-stage nonrevealing value functions and (2) constructing strategies by blocks in order to link the values of the nonrevealing games with the original values.

RSS-based joint detection and tracking in mixed LOS and NLOS environments

This paper considers the problem of joint detection and tracking in mixed line-of-sight (LOS) and non-line-of-sight (NLOS) environments by using received signal strength (RSS) measurements. A nonlinear target tracking model with multiple switching parameters has been proposed, in which multiple independent Markov chains are used to describe the switching of target maneuvers and the transition of LOS/NLOS measurements, respectively. Based on the proposed tracking model, a multi-sensor multiple model Bernoulli filter (MMBF) has been developed by employing the random finite set theory which can formulate the joint detection and tracking in a unified framework. To derive a closed-form expression to the MMBF, the Gaussian mixture implementations have been provided by applying the extended Kalman filter technique. A numerical example is provided involving tracking a maneuvering target by a sensor network with 30 nodes. Simulation results confirm the effectiveness of the proposed filter.

Mortality modelling with regime-switching for the valuation of a guaranteed annuity option

We consider three ways of putting forward a regime-switching approach in modelling the evolution of mortality rates for the purpose of pricing a guaranteed annuity option (GAO). This involves the extension of the Gompertz and non-mean reverting models as well as the adoption of a pure Markov model for the force of mortality. A continuous-time finite-state Markov chain is employed to describe the evolution of mortality model parameters which are then estimated using the filtered-based and least-squares methods. The adequacy of the regime-switching Gompertz model for the US mortality data is demonstrated via the goodness-of-fit metrics and likelihood-based selection criteria. A GAO is valued assuming the interest and mortality risk factors are switching regimes in accordance with the dynamics of two independent Markov chains. To obtain closed-form valuation formulae, we employ the change of measure technique with the pure endowment price as the numéraire. Numerical implementations are included to compare the results of the proposed approaches and those from the Monte Carlo simulations.

Parallel Markov chain Monte Carlo for non‐Gaussian posterior distributions

Recent developments in big data and analytics research have produced an abundance of large data sets that are too big to be analysed in their entirety, because of limits on computer memory or storage capacity. To address these issues, communication‐free parallel Markov chain Monte Carlo methods have been developed for Bayesian analysis of big data. These methods partition data into manageable subsets, perform independent Bayesian Markov chain Monte Carlo analysis on each subset and combine the subset posterior samples to estimate the full data posterior. Current approaches to combining subset posterior samples include sample averaging, weighted averaging and kernel smoothing techniques. Although these methods work well for Gaussian posteriors, they are not well suited to non‐Gaussian posterior distributions. Here, we develop a new direct density product method for combining subset marginal posterior samples to estimate full data marginal posteriors. Using commonly implemented distance metrics, we show in simulation studies of Bayesian models with non‐Gaussian posteriors that our method outperforms the existing methods in approximating the full data marginal posteriors. Because our method estimates only marginal densities, there is no limitation on the number of model parameters analysed. Our procedure is suitable for Bayesian models with unknown parameters with fixed dimension in continuous parameter spaces. Copyright © 2015 John Wiley & Sons, Ltd.

H<SUB align="right">∞ output feedback control of networked control systems with multiple quantisers

H∞ output feedback control of a kind of multi-input and multi-output networked control system (NCS) with packet loss processes is considered in this paper, where the packet loss processes are modelled by independent Markov chains. Multiple quantisers are located at both of the sensor nodes and the controller nodes. For each quantiser, the quantisation density varies with the successive packet dropout number that the corresponding quantised signal experiences. A sufficient condition of the exponential stability with a prescribed H∞ performance level is developed based on the Lyapunov-Krasovskii function approach. Then, a quantised dynamic output feedback controller is designed via a cone complement linearisation method. Finally, a simulation result is provided to demonstrate the effectiveness of the designed method.

Prevalence of Theileria equi and Babesia caballi as well as the identification of associated ticks in sympatric Grevy's zebras (Equus grevyi) and donkeys (Equus africanus asinus) in northern Kenya.

The role of equine piroplasmosis as a factor in the population decline of the Grevy's zebra is not known. We determined the prevalence of Babesia caballi and Theileria equi in cograzing Grevy's zebras (Equus grevyi) and donkeys (Equus africanus asinus) in northern Kenya and identified the associated tick vectors. Blood samples were taken from 71 donkeys and 16 Grevy's zebras from March to May 2011. A nested PCR reaction using 18s ribosomal (r)RNA primers on 87 blood spots showed 72% (51/71; 95% confidence interval [CI] 60.4-81.0%) of donkeys and 100% (16/16; 95% CI, 77.3-100%) of Grevy's zebras were T. equi positive. No samples were positive for B. caballi. Sequence comparison using the National Center for Biotechnology Information's basic local alignment search tool identified homologous 18s rRNA sequences with a global geographic spread. The T. equi-derived sequences were evaluated using Bayesian approaches with independent Metropolis-coupled Markov chain Monte Carlo runs. The sequences clustered with those found in Sudan, Croatia, Mongolia, and the US, with statistical support greater than 80% for the two main clades. Hyalomma tick species were found on both donkeys and Grevy's zebras, whereas Rhipicephalus pulchellus was found exclusively on Grevy's zebras and Hyalomma marginatum rupfipes on donkeys. The prevalence of T. equi was 100% in Grevy's zebras and 72% in donkeys with common tick vectors identified. Our results suggest that donkeys and Grevy's zebras can be asymptomatic carriers and that piroplasmosis is endemic in the study area.

Occupancy grid map of semi-static objects by mobile observer

In this research we propose a method for spatial representation of semi-static objects in dynamic environment. For this objective, we extend IMAC (Independent Markov Chain Occupancy) model, which is a map constructing method for dynamic environment. In the IMAC model, a mobile observer cannot estimate the grid map correctly if it does not stop enough time at the same location. To overcome this limitation, we propose a technique to estimate the IMAC grid map from an analytical discussion by simulating a grid-world model for a simple straight line path. The simulation result shows that if the number of observations for a given grid cell is sufficient, both parameters of the two transition probability parameters of IMAC model can be estimated correctly. We carried out a simple test in real-world environment using a vehicle. The experimental results show that we can create a map from GPS data recorded by a smartphone installed on a vehicle. The contributions of this paper are as follows: (1) propose a new technique for estimating Poisson parameters for IMAC grid map in dynamic environment; (2) prove the model can be used in real world.

Reliable H∞ Control for a Class of Nonlinear Networked Control Systems with Random Delays

AbstractIn this paper, the problem of reliable H∞ controller design is studied for a class of nonlinear networked control systems. A novel model is presented, which contains random transmission delays, faults of the sensor and actuator. The sensor‐to‐controller and controller‐to‐actuator transmission delays with upper bounds are considered, simultaneously. The working conditions of the sensor and actuator are formulated as two independent Markov chains, which take matrix values in finite sets, respectively. The resulting closed‐loop system is converted into a Markov switching system. On the basis of the cone complementary linearization algorithm, a mode‐dependent reliable controller is constructed such that the closed‐loop system is stochastically stable and attains the prescribed H∞ disturbance attenuation level. Finally, a numerical example is given to show the effectiveness of the developed technique.

Consensus in Markovian jump second‐order multi‐agent systems with random communication delay

In this study, the consensus problem for a class of second-order multi-agent systems (MASs) with Markovian characterisations is investigated. The stochastic switching topology and the random communication delay are dominated by two mutually independent Markov chains. The communication delay exists in the switching signal as well as the position information exchanges in the authors’ work. A novel consensus protocol is presented without using the neighbours’ velocity information. By performing three steps of model transformation and introducing a mapping for the two independent Markov chains, the original system is converted into an expanded analogous error system with two Markovian jumping parameters. A necessary and sufficient criterion for the mean square consensus of the Markovian jump second-order MASs with random communication delay is derived. Finally, a numerical example is given to illustrate the feasibility and effectiveness of the theoretical result.

Robust mixed [formula omitted] control of networked control systems with random delays and partially known transition matrix

A robust two-mode-dependent controller is discussed in this paper for a special jump linear networked control systems. Delays of random nature exist in both measurement channel (C/A) and actuation channel (S/C). These delays are modeled through two homogeneous independent Markov chains. Terminologies for the norms of H2 and H∞ for such systems are also given. The two mode dependent output feedback control law is proposed, which means that its value depends on the current delay existing in measurement channel and also on the previous delay in actuation channel. Further more, a robust H2 and robust mixed H2/H∞ controllers are developed for networked control systems (NCSs). Similar analyses and design procedures are also carried out for the case of partially known transition probability matrices. In the end, a numerical example is presented and comparison is made between the results obtained in this paper and the stability criteria developed by Zong et al. [11] to illustrate the effectiveness of the developed methodology.

ParallelMCMCcombine: an R package for bayesian methods for big data and analytics.

Recent advances in big data and analytics research have provided a wealth of large data sets that are too big to be analyzed in their entirety, due to restrictions on computer memory or storage size. New Bayesian methods have been developed for data sets that are large only due to large sample sizes. These methods partition big data sets into subsets and perform independent Bayesian Markov chain Monte Carlo analyses on the subsets. The methods then combine the independent subset posterior samples to estimate a posterior density given the full data set. These approaches were shown to be effective for Bayesian models including logistic regression models, Gaussian mixture models and hierarchical models. Here, we introduce the R package parallelMCMCcombine which carries out four of these techniques for combining independent subset posterior samples. We illustrate each of the methods using a Bayesian logistic regression model for simulation data and a Bayesian Gamma model for real data; we also demonstrate features and capabilities of the R package. The package assumes the user has carried out the Bayesian analysis and has produced the independent subposterior samples outside of the package. The methods are primarily suited to models with unknown parameters of fixed dimension that exist in continuous parameter spaces. We envision this tool will allow researchers to explore the various methods for their specific applications and will assist future progress in this rapidly developing field.

Robust and accelerated Bayesian inversion of marine controlled-source electromagnetic data using parallel tempering

Bayesian methods can quantify the model uncertainty that is inherent in inversion of highly nonlinear geophysical problems. In this approach, a model likelihood function based on knowledge of the data noise statistics is used to sample the posterior model distribution, which conveys information on the resolvability of the model parameters. Because these distributions are multidimensional and nonlinear, we used Markov chain Monte Carlo methods for highly efficient sampling. Because a single Markov chain can become stuck in a local probability mode, we run various randomized Markov chains independently. To some extent, this problem can be mitigated by running independent Markov chains, but unless a very large number of chains are run, biased results may be obtained. We got around these limitations by running parallel, interacting Markov chains with “annealed” or “tempered” likelihoods, which enable the whole system of chains to effectively escape local probability maxima. We tested this approach using a transdimensional algorithm, where the number of model parameters as well as the parameters themselves were treated as unknowns during the inversion. This gave us a measure of uncertainty that was independent of any particular parameterization. We then subset the ensemble of inversion models to either reduce uncertainty based on a priori constraints or to examine the probability of various geologic scenarios. We demonstrated our algorithms’ fast convergence to the posterior model distribution with a synthetic 1D marine controlled-source electromagnetic data example. The speed up gained from this new approach will facilitate the practical implementation of future 2D and 3D Bayesian inversions, where the cost of each forward evaluation is significantly more expensive than for the 1D case.

Dynamic output feedback of networked control systems with partially known Markov chain packet dropouts

SummaryIn this paper, a class of networked control systems with output feedback control and H ∞ performance is discussed. It considers packet dropouts in both measurement (S/C) and actuation (C/A) channels. Markovian chain principle is used in modeling the packet dropouts in S/C and C/A channels. The time scale adopted in these two independent homogeneous Markov chain is linear with the physical time. The model also takes into consideration the late arriving packets. The effect of interaction between packet dropouts in both channels on the stability of system, when the networks of both (S/C) and (C/A) channels overlap is also examined. Sufficient condition for the existence of H ∞ output feedback controller is presented and it is shown that it is dependent on the upper bounds of the number of consecutive packet dropouts. The developed stability analysis and control scheme is also investigated with partially known transition probability matrices. Finally, a numerical example has been given to show the effectiveness of developed method. Copyright © 2013 John Wiley & Sons, Ltd.

Detection of Significant Genomic Alterations via Simultaneous Minimal Sojourns at a State by Independent Continuous-time Markov Chains

The connection between a disease and some genetic disorders can be studied by looking for recurrent alterations in genomic profiles of humans affected by the disease. Modeling each patient profile as a 2-state jump Markov process, we define a recurrent alteration in terms of a sojourn time of a cumulated process. The cumulated process happen to be a birth-and-death process. We provide the exact Laplace transforms of relevant hitting times associated with the latter. Based on this exact results, we derive an upper bound of a sojourn time probability of interest. Our upper bound is exactly equal to the p-value nominated in all earlier literature where an attempt is made for an assessment of the significance of genomic alterations. Moreover, that p-value has never been evaluated exactly in previous works. Finally, we give abaques which are useful for the assessment of recurrent genomic alterations for a variety of realistic alteration frequencies and lengths.

Distributed multiple model estimation for jump Markov linear systems with missing measurements

This paper is concerned with distributed multiple model estimation for jump Markov linear systems with missing measurements over a sensor network. Two independent Markov chains are used to describe the switching of dynamic models and the missing of measurements, respectively. Under the assumption that each sensor can only communicate with its neighbours, a distributed filter is developed by applying the basic interacting multiple model (IMM) approach in the Bayesian estimation framework. To circumvent the difficulty of exponentially growing filters by exchanging local measurements between neighbouring sensors, the mode-conditioned estimates are exchanged instead of local measurements and the covariance intersection method is adopted to fuse mode-conditioned estimates. A multi-sensor manoeuvering target tracking example is provided to verify the effectiveness of the proposed filter.

Distributed Multiple-Model Estimation for Simultaneous Localization and Tracking With NLOS Mitigation

This paper studies the problem of simultaneous localization and tracking (SLAT) in non-line-of-sight (NLOS) environments. By combining a target state and a sensor node location into an augmented vector, a nonlinear system with two jumping parameters is formulated in which two independent Markov chains are used to describe the switching of the target maneuvers and the transition of LOS/NLOS, respectively. To derive the state estimate of the proposed jump Markov nonlinear system for each sensor node, an interacting multiple-model (IMM) approach and a cubature Kalman filter (CKF) are employed. As the number of mode-conditioned filters exponentially grows with the increases in the number of active sensor nodes in the centralized fusion, a distributed scheme is adopted to reduce the computational burden, and a covariance intersection (CI) method is used to fuse sensor-based target-state estimates. A numerical example is provided, involving tracking a maneuvering target by a set of sensors, and simulation results show that the proposed filter can track the target and can estimate the positions of active sensor nodes accurately.

Stability analysis of networked control systems with round-robin scheduling and packet dropouts

In this paper, the stability of networked control systems (NCSs) with communication constraints at both channels is investigated. A Conventional Round-Robin Scheduling (CRRS) is applied to deal with the communication constraints issue for its simple structure. Furthermore, a Dynamic Round-Robin Scheduling (DRRS), which can preserve the controllability and the detectability of the systems, is considered. For the unreliable communication channels, two independent homogeneous Markov chains are selected to model the packet dropouts phenomenon in the sensor-to-controller (S/C) channel and the controller-to-actuator (C/A) channel. According to the periodic property of the Round-Robin Scheduling (RRS), an auxiliary system with augmented Markov chain is established by the lifting technique to facilitate the stability analysis of the closed-loop system. A necessary and sufficient condition of the exponential mean-square stability for the NCSs is derived. Two illustrative examples are shown to demonstrate the effectiveness of the proposed stability analysis method.