Time-homogeneous Markov Chain Research Articles

The Recurrence and Transience of Random Walks

This article reviews the research history and application fields of random walks and abstracts random walks into a time-homogeneous Markov chain to study their recurrent and transient properties. For one-dimensional and two-dimensional random walks, the likelihood of returning in steps and the probability of returning for the first time in steps of each state are first introduced, along with their relationship. Then the Stirling's formula is given, which is utilized to estimate the probability of returning in n steps, and the convergence and divergence of infinite series is used to prove that random walk in one dimension is recurrent. Random walk in two dimension has similar properties, which is a bit more complicated by using the relation between polynomials. Higher dimensional random walks need to consider special states first, and then generalize them to other states. Finally, this paper concluded that one-dimensional and two-dimensional random walks are recurrent, and random walks with dimensions higher than two are transient.

High-frequency stock market order transitions during the US-China trade war 2018: A discrete-time Markov chain analysis.

Statistical analysis of high-frequency stock market order transaction data is conducted to understand order transition dynamics. We employ a first-order time-homogeneous discrete-time Markov chain model to the sequence of orders of stocks belonging to six different sectors during the US-China trade war of 2018. The Markov property of the order sequence is validated by the Chi-square test. We estimate the transition probability matrix of the sequence using maximum likelihood estimation. From the heatmap of these matrices, we found the presence of active participation by different types of traders during high volatility days. On such days, these traders place limit orders primarily with the intention of deleting the majority of them to influence the market. These findings are supported by high stationary distribution and low mean recurrence values of add and delete orders. Further, we found similar spectral gap and entropy rate values, which indicates that similar trading strategies are employed on both high and low volatility days during the trade war. Among all the sectors considered in this study, we observe that there is a recurring pattern of full execution orders in the Finance & Banking sector. This shows that the banking stocks are resilient during the trade war. Hence, this study may be useful in understanding stock market order dynamics and devise trading strategies accordingly on high and low volatility days during extreme macroeconomic events.

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.

Equivalences of Geometric Ergodicity of Markov Chains

This paper gathers together different conditions which are all equivalent to geometric ergodicity of time-homogeneous Markov chains on general state spaces. A total of 34 different conditions are presented (27 for general chains plus 7 for reversible chains), some old and some new, in terms of such notions as convergence bounds, drift conditions, spectral properties, etc., with different assumptions about the distance metric used, finiteness of function moments, initial distribution, uniformity of bounds, and more. Proofs of the connections between different conditions are provided, somewhat self-contained but using some results from the literature where appropriate.

Sampling-Dependent Transition Paths of Iceland–Scotland Overflow Water

Abstract In this note, we apply transition path theory (TPT) from Markov chains to shed light on the problem of Iceland–Scotland Overflow Water (ISOW) equatorward export. A recent analysis of observed trajectories of submerged floats demanded revision of the traditional abyssal circulation theory, which postulates that ISOW should steadily flow along a deep boundary current (DBC) around the subpolar North Atlantic prior to exiting it. The TPT analyses carried out here allow attention to be focused on the portions of flow from the origin of ISOW to the region where ISOW exits the subpolar North Atlantic and suggest that insufficient sampling may be biasing the aforementioned demand. The analyses, appropriately adapted to represent a continuous input of ISOW, are carried out on three time-homogeneous Markov chains modeling the ISOW flow. One is constructed using a high number of simulated trajectories homogeneously covering the flow domain. The other two use much fewer trajectories which heterogeneously cover the domain. The trajectories in the latter two chains are observed trajectories or simulated trajectories subsampled at the observed frequency. While the densely sampled chain supports a well-defined DBC, whether this is a peculiarity of the simulation considered or not, the more heterogeneously sampled chains do not, irrespective of the nature of the trajectories used, i.e., observed or simulated. Studying the sampling sensitivity of the Markov chains, we can give recommendations for enlarging the existing float dataset to improve the significance of conclusions about long-time-asymptotic aspects of the ISOW circulation.

Estimating MS-BLGARCH Models Using Recursive Method

In this paper a new class of models is proposed for modeling nonlinear and stationary time series. This new class of models is referred to as the Markov-switching bilinear GARCH (MS-BLGARCH) models. In these models, the parameters are allowed to depend on an unobservable time-homogeneous and stationary Markov chain with finite state space. The statistical inference for these models is rather difficult due to the dependence on the whole regime path. We propose a recursive algorithm for parameter estimation in MS − BLGARCH. The proposed method is useful for long time series as well as for data available in real time. The main idea is to use the maximum likelihood estimation (MLE) method and from this develop a recursive Expectation-Maximization (EM) algorithm.

A DOUBLY MARKOV SWITCHING AR MODEL: SOME PROBABILISTIC PROPERTIES AND STRONG CONSISTENCY

In this work, we consider doubly Markov switching AR models, where analytic tractability and flexibility are quite simply a competitive advantage, which becomes an attractive tool for modeling economic and financial time series. In these models, the parameters are allowed to depend on an unobservable time-homogeneous Markov chain with finite state space. So, we discuss some basic probabilistic properties of $$D-MSAR\left( p,q\right)$$ model such as conditions ensuring the existence of strict, second-order stationarity solution, causal and ergodic solution and its moments properties. The quasi-maximum likelihood estimator of the parameters in the model is shown to be strongly consistent.

한국 신용등급 전이행렬에 대한 시계열성 연구

Many articles offer credit transition matrix is modelled by time-homogeneous Markov Chain. However, some articles in overseas provide the existence of time series patterns and relationship with economic situations in ratings changes. No empirical study regarding systematic approaches for time series analysis among articles related ratings changes in Korea, and that is a reason we conduct this article. This article uses time series data are credit ratings migrations matrices of all three ratings agencies in Korea from 1998 to 2020. We find time series patterns in most of ratings changes, especially rating downgrades changes. Additionally, we propose a time series model that is an autoregressive model and estimate cumulative default rates using a model for ratings changes. Our model shows a better estimated result compared to a simple Markov model.

On nonrecurrence of nonlinear random time delay autoregressive models under random environment

In this paper, random environment and random time delay are introduced into general nonlinear autoregressive models to improve the applicability of these models. The introduction of random environment makes nonlinear autoregressive models more flexible and more widely used. Random time delay is a new way for models to be applied in new fields. In order to analyze the properties of the proposed new model, we construct a new stochastic process and prove that it is an irreducible and aperiodic time-homogeneous Markov chain. The nonrecurrence of the proposed new type of models is considered, and some sufficient conditions for nonrecurrence are investigated.

Dynamical geography and transition paths of Sargassum in the tropical Atlantic

By analyzing a time-homogeneous Markov chain constructed using trajectories of undrogued drifting buoys from the NOAA Global Drifter Program, we find that probability density can distribute in a manner that resembles very closely the recently observed recurrent belt of high Sargassum concentration in the tropical Atlantic between 5 and 10°N, coined the Great Atlantic Sargassum Belt (GASB). A spectral analysis of the associated transition matrix further unveils a forward attracting almost-invariant set in the northwestern Gulf of Mexico with a corresponding basin of attraction weakly connected with the Sargasso Sea but including the nutrient-rich regions around the Amazon and Orinoco rivers mouths and also the upwelling system off the northern coast of West Africa. This represents a data-based inference of potential remote sources of Sargassum recurrently invading the Intra-Americas Seas (IAS). By further applying Transition Path Theory (TPT) to the data-derived Markov chain model, two potential pathways for Sargassum into the IAS from the upwelling system off the coast of Africa are revealed. One TPT-inferred pathway takes place along the GASB. The second pathway is more southern and slower, first going through the Gulf of Guinea, then across the tropical Atlantic toward the mouth of the Amazon River, and finally along the northeastern South American margin. The existence of such a southern TPT-inferred pathway may have consequences for bloom stimulation by nutrients from river runoff.

Robust distributed adaptive consensus for linear multiagent systems with uncertain topologies

This paper focuses on the robust mean-square consensus control problem for linear multiagent systems over randomly switching signed interaction topologies. The stochastic process is governed by a time-homogeneous Markov chain with partly unknown transition rates. Sufficient conditions for a consensus in the form of linear matrix inequalities are given via distributed adaptive control based on parameter-dependent Lyapunov functions. The adaptive control protocols require only the neighbor information of the agents, and the algorithm that designs the protocols reduces the influence of the communication topology on the consensus, which can prevent undesirable interaction impacts. Moreover, the disturbance rejection problem is addressed as an extension. Finally, two simulations are utilized to illustrate the effectiveness of the algorithms.

Output-Feedback Control Under Hidden Markov Analog Fading and Redundant Channels

This brief investigates the output-feedback control problem with redundant channels, in which the output measurements may suffer from the Markov analog fading when transmitting through the independent primary channel and redundant channels. The actual channel mode obeys a two-state time-homogeneous Markov chain, and the expectation of fading amplitude in the good mode is much higher than the one in the bad mode. The mode detectors are employed to estimate the status of both the primary and redundant channels, by which the better measurement signals may be selected for the controller. Thus, a token-dependent output-feedback controller is constructed, and then, the stability condition is obtained via the actual mode-dependent Lyapunov function method. Finally, the proposed output-feedback control scheme is applied to an operational amplifier (OPA) circuit under the hidden Markov analog fading channels.

An Analysis of a Storage System for a Wind Farm with Ramp-Rate Limitation

This paper provides evidence on how the variability of the power produced by a wind farm and its revenue are affected by implementing a ramp-rate limitation strategy and by adding a storage device to the system. The wind farm receives penalties whenever the ramp-rate limitations are not respected and may be supported by batteries to avoid this scenario. In this paper, we model the battery usage as a discrete time homogeneous Markov chain with rewards thanks to which it is possible to simulate the state of the charge of the battery and to calculate the amount of penalties suffered by the wind farm during any period. An application is performed considering the power produced by a hypothetical wind turbine located in Sardinia (Italy) using real wind speed data and electricity prices from a period of 10 years. We applied the concept of ramp-rate limitation on our hourly dataset, studying several limitation scenarios and battery capacities.

Kinetic uncertainty relation on first-passage time for accumulated current.

The kinetic uncertainty relation (KUR) is a trade-off relation between the precision of an observable and the mean dynamical activity in a fixed time interval for a time-homogeneous and continuous-time Markov chain. In this Letter, we derive the KUR on the first passage time for the time-integrated current from the information inequality at stopping times. The relation shows that the precision of the first passage time is bounded from above by the mean number of jumps up to that time. We apply our result to simple systems and demonstrate that the activity constraint gives a tighter bound than the thermodynamic uncertainty relation in the regime far from equilibrium.

Robust distributed adaptive consensus for discrete-time multiagent systems with uncertain topologies

This paper considers the stochastic consensus problem of high-order linear discrete-time multiagent systems (MASs) with uncertain topologies and external disturbances. The randomly switching interaction signed topologies are governed by a time-homogenous Markov chain with incomplete knowledge of the transition probabilities. Based on Lyapunov function theory and adaptive control strategies, we employ a homogeneous parameter-dependent Lyapunov function to propose sufficient conditions under a set of linear matrix inequality (LMI) constraints. A free-weighting matrix method is introduced to give the proposed LMIs extra freedom to search feasible solutions. Furthermore, the adaptive controllers applied in this paper are independent of topologies and fully distributed; i.e., the system achieves robust consensus without continuously updating the controller and dealing with global data. Additionally, we extend the theoretical framework to analyze the disturbance rejection performance of MASs under the stochastic process. Two numerical examples are provided to show the effectiveness of the algorithms.

Transition paths of marine debris and the stability of the garbage patches.

We used transition path theory (TPT) to infer "reactive" pathways of floating marine debris trajectories. The TPT analysis was applied on a pollution-aware time-homogeneous Markov chain model constructed from trajectories produced by satellite-tracked undrogued buoys from the National Oceanic and Atmospheric Administration's Global Drifter Program. The latter involved coping with the openness of the system in physical space, which further required an adaptation of the standard TPT setting. Directly connecting pollution sources along coastlines with garbage patches of varied strengths, the unveiled reactive pollution routes represent alternative targets for ocean cleanup efforts. Among our specific findings we highlight: constraining a highly probable pollution source for the Great Pacific garbage patch; characterizing the weakness of the Indian Ocean gyre as a trap for plastic waste; and unveiling a tendency of the subtropical gyres to export garbage toward the coastlines rather than to other gyres in the event of anomalously intense winds.

Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time

A new definition of continuous-time equilibrium controls is introduced. As opposed to the standard definition, which involves a derivative-type operation, the new definition parallels how a discrete-time equilibrium is defined and allows for unambiguous economic interpretation. The terms “strong equilibria” and “weak equilibria” are coined for controls under the new and standard definitions, respectively. When the state process is a time-homogeneous continuous-time Markov chain, a careful asymptotic analysis gives complete characterizations of weak and strong equilibria. Thanks to the Kakutani–Fan fixed-point theorem, the general existence of weak and strong equilibria is also established under an additional compactness assumption. Our theoretic results are applied to a two-state model under nonexponential discounting. In particular, we demonstrate explicitly that there can be incentive to deviate from a weak equilibrium, which justifies the need for strong equilibria. Our analysis also provides new results for the existence and characterization of discrete-time equilibria under infinite horizon.

Application of Markov renewal theory and semi‐Markov decision processes in maintenance modeling and optimization of multi‐unit systems

AbstractIn this paper, a condition‐based maintenance model for a multi‐unit production system is proposed and analyzed using Markov renewal theory. The units of the system are subject to gradual deterioration, and the gradual deterioration process of each unit is described by a three‐state continuous time homogeneous Markov chain with two working states and a failure state. The production rate of the system is influenced by the deterioration process and the demand is constant. The states of the units are observable through regular inspections and the decision to perform maintenance depends on the number of units in each state. The objective is to obtain the steady‐state characteristics and the formula for the long‐run average cost for the controlled system. The optimal policy is obtained using a dynamic programming algorithm. The result is validated using a semi‐Markov decision process formulation and the policy iteration algorithm. Moreover, an analytical expression is obtained for the calculation of the mean time to initiate maintenance using the first passage time theory.

A Quantitative Resilience Measure Framework for Power Systems Against Wide-Area Extreme Events

A framework for quantitative evaluation of the power system resilience considering multimicrogids effects is proposed in this article. The main contribution of the proposed approach is that it incorporates various features underlined in the concept of the power system resilience, such as loss of load probability, expected demand not supplied, system fragility, system recovery difficulties, and system adaption ability after experiencing a destructive event. In addition to the power system aspects, the proposed resilience framework employs the type and severity of natural disasters to achieve realistic results. To model the impact of multiple-microgrids, at first, the discrete-time multistate transition model of the power system under an extreme event is obtained. Then the probability of system states (normal, microgrid, and emergency) is calculated using the time-independent transition matrix and according to the time-homogeneous Markov chain. The effectiveness of the proposed method is tested on the IEEE 30-bus test case and real electricity network of Great Britain in five episodes each with 2000 scenarios.

Symbolic Time Series Analysis for Anomaly Detection in Measure-Invariant Ergodic Systems

Abstract This paper presents a novel framework of symbolic time series analysis (STSA) for anomaly detection in dynamical systems. The core concept is built upon a property of measure-preserving transformation (MPT) sequence, acting on a probability space with ergodic measure, that the eigenfunctions of these transformations would be time-invariant. As a result, unlike a standard STSA that is required to generate time-homogeneous Markov chains, the proposed MPT-based STSA is allowed to have time-inhomogeneous Markov chains, where the (possibly time-varying) state transition probability matrices have time-invariant eigenvectors. Such a time-invariance facilitates analysis of the dynamical system by using short-length time series of measurements. This is particularly important in applications, where the underlying dynamics and process anomalies need fast monitoring and control actions in order to mitigate any potential structural damage and/or to avoid catastrophic failures. The MPT-based STSA has been applied for low-delay detection of fatigue damage, which is a common source of failures in mechanical structures and which is known to have uncertain dynamical characteristics. The underlying algorithm has been validated with experimental data generated from a laboratory apparatus that uses ultrasonic sensors to detect fatigue damage in polycrystalline–alloy specimens. The performance of the proposed MPT-based STSA is evaluated by comparison with those of a standard STSA and a hidden Markov model (HMM) on the same experimental data. The results consistently show superior performance of the MPT-based STSA.