Time-inhomogeneous Markov Chains Research Articles

Embedding of Markov matrices for d⩽4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varvec{d \\leqslant 4}$$\\end{document}

The embedding problem of Markov matrices in Markov semigroups is a classic problem that regained a lot of impetus and activities through recent needs in phylogeny and population genetics. Here, we give an account for dimensions d⩽4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d\\leqslant 4$$\\end{document}, including a complete and simplified treatment of the case d=3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=3$$\\end{document}, and derive the results in a systematic fashion, with an eye on the potential applications. Further, we reconsider the setup of the corresponding problem for time-inhomogeneous Markov chains, which is needed for real-world applications because transition rates need not be constant over time. Additional cases of this more general embedding occur for any d⩾3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d\\geqslant 3$$\\end{document}. We review the known case of d=3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=3$$\\end{document} and describe the setting for future work on d=4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=4$$\\end{document}.

Assessment on fast simulation of wind-driven pollutant dispersion around a street canyon with regime-switching Markov chain

High-density cities shorten commutes but also create proximity to air pollution sources. Sensors coupled with simulations can achieve real-time monitoring and risk assessment, and the recently emerging data-driven models provide opportunities that allow minimizing the computation burden of physical models. This study investigates the use of a time inhomogeneous Markov chain (CLF-MC) to fast simulation of wind-driven pollutant dispersion around a street canyon. The CLF-MC model is trained and tested using synthetic datasets obtained from large eddy simulations (LES). The LES model is validated using a benchmark wind tunnel test, which yields correlation coefficients above 0.95. The results show that the CLF-MC can faithfully reproduce the cumulative distribution of air pollution concentrations at the receptor site using the optimal model configuration identified in this study. For the hold-out test dataset, the release condition is a time-varying release strength, and the CLF-MC shows improved prediction results compared to the baseline models, i.e. the weighted average deviation is 15 % and 51 % smaller, respectively. In comparison to LES, the CLF-MC model is 4.8 × 106 times faster. These findings provide scientific references for fast and accurate wind-driven pollutant dispersion simulation using the probabilistic modelling approach.

Polynomial Recurrence of Time-inhomogeneous Markov Chains

This paper is devoted to establishing conditions that guarantee the existence of a p-th moment of the time it takes for a timeinhomogeneous Markov chain to hit some set C. We modified the well-known Drift Condition from the theory of homogeneous Markov chains. We demonstrated that the inhomogeneous Markov chain may be polynomially recurrent while exhibiting different dynamics from its homogeneous counterpart.

Integrated Color and Intensity MSF Scheme for Image Retrieval Systems

Markov Stationary Features (MSF) based on homogeneous Markov chain model, for content-based image analysis, is getting popularity nowadays. It not only considers the distribution of colors, just as the histogram method does, but also characterizes the spatial co-occurrence of histogram patterns. However, handling a large-scale database of images with a degree of heterogeneity, a simple MSF method is not sufficient to discriminate the images as one requires. In this paper, an Integrated Color and Intensity MSF (ICI-MSF) based on non-homogeneous Markov Chain is proposed to overcome this shortcoming. By incorporating spatial co-occurrence of image intensities with the spatial co-occurrence information of colors and exploiting time inhomogeneous Markov chain concept, it is possible to improve certain aspects of the existing methods. Without compromising effectiveness and robustness, the method proposed in this paper keeps the feature level simplicity. Widely recognized databases namely WANG1000 and Corel10800, are used to evaluate and compare the performance of the proposed algorithm with the existing methods. The experimental results justify the effectiveness of the proposed method.

Stochastic simulation of occupant-driven energy use in a bottom-up residential building stock model

The residential buildings sector is one of the largest electricity consumers worldwide and contributes disproportionally to peak electricity demand in many regions. Strongly driven by occupant activities, household energy consumption is stochastic and heterogeneous in nature. However, most residential energy models applied by industry use homogeneous, deterministic activity schedules, which work well for predictions of annual energy consumption, but can result in unrealistic hourly or sub-hourly electric load profiles, with exaggerated or muted peaks. The increasing proportion of variable renewable energy generators means that representing the heterogeneity and stochasticity of occupant behavior is now crucial for reliable planning at both bulk-power and distribution-system scales. This work presents a novel and open-source occupancy simulation approach that can simulate a diverse set of individual occupant and household event schedules for all major electricity, fuel, and hot water end uses. To accomplish this, we evaluated three alternative occupant activity simulation approaches before selecting a hybrid combining time-inhomogeneous Markov chains and probability-sampling of event durations and magnitudes. We integrated the stochastic occupancy simulation with an open-source bottom-up physics-simulation building stock model and published a set of 550,000 diverse household end-use activity schedules representing a national housing stock. The simulator was verified against time-use survey data, and simulation results were validated against measured end-use electricity data for accuracy and reliability. While we use data for the United States, our application demonstrates how similar approaches could be applied using the time-use survey data collected in many countries around the world.

Simulating Portfolio Decisions under Uncertainty When the Risky Asset and Short Rate Are Modulated by an Inhomogeneous and Asset-Dependent Markov Chain

This paper aims to simulate portfolio decisions under uncertainty when the diffusion parameters of the risky asset and short rate paid for a bond are both modulated by a time-inhomogeneous Markov chain, with transition probabilities dependent on states, time, and asset prices. To do this, we first found closed-form solutions of the corresponding utility-maximization problem, which solves a rational consumer that makes portfolio and consumption decisions by using the corresponding infinitesimal generator associated with the Markov chain. Subsequently, as an illustration of the theoretical results obtained, several scenarios were simulated for the Mexican case. The expected economic policy was inferred from announced monetary policy decisions regarding the reference rate and possible changes in trend due to the lack of fiscal stimuli. Under this framework, states were defined from the current and expected economic policies, and transition probabilities were expressed in terms of the ratio between the prices of the risky asset and the bond. It should be noted, as far as the authors know, that no analytical solutions are known in the literature for the case of Markov-modulated time-inhomogeneous chains with transition probabilities that, themselves, are stochastic processes.

Analysis of Exchange Rates as Time-Inhomogeneous Markov Chain with Finite States

Irrespective of whether the test for homogeneity is significant or not, most researchers assume time-homogeneity in analysing Markov chains due to scanty literature on the analysis of time-inhomogeneous Markov chains. Based on the assumption that, for each point in time in the future, a stochastic process will be subjected to a randomly selected transition matrix from an ergodic set of transition matrices the process was subjected to in the recent past, a methodology was proposed for analysing the long-run behaviours of time-inhomogeneous Markov chains. The proposed model was implemented to historical data consisting of the exchange rate of cedi-dollar, cedi-pound, and cedi-euro spanning over 6 years (January 2012 to December 2017). The results show that under certain “closeness” conditions, the long-run behaviours of the time-inhomogeneous case are almost identical to those of the time-homogeneous case. The paper asserted that even if the Markov chain exhibit time-inhomogeneity, analysing the Markov chain under the assumption of time-homogeneity is a step in the right direction under certain “closeness” conditions; otherwise, the proposed method is recommended. It was also found that investing in dollars yields better returns than the other currencies in Ghana.

A Time-Inhomogeneous Prendiville Model with Failures and Repairs

We consider a time-inhomogeneous Markov chain with a finite state-space which models a system in which failures and repairs can occur at random time instants. The system starts from any state j (operating, F, R). Due to a failure, a transition from an operating state to F occurs after which a repair is required, so that a transition leads to the state R. Subsequently, there is a restore phase, after which the system restarts from one of the operating states. In particular, we assume that the intensity functions of failures, repairs and restores are proportional and that the birth-death process that models the system is a time-inhomogeneous Prendiville process.

Inventory management for the in-flight catering industry: A case of uncertain demand and product substitutability

In-flight caterers are faced with an immense challenge of catering for each passenger on-board a flight without knowing the flight’s final passenger load or the meal preferences of its passengers. These uncertainties lead to excessive amounts of waste generation because in-flight caterers tend to over-produce meals to mitigate the risk of meal shortages and dissatisfied passengers. This paper evaluates the value of combining product substitution and meal demand uncertainty within an inventory decision-making model as a potential solution opportunity for the wastage dilemma faced by the in-flight catering industry. The model developed is defined as a stochastic and multi-objective mixed-integer programming model with fixed recourse and static product substitution. The model relies on the output of a time-inhomogeneous Markov Chain forecasting model with a multiple regression analysis to forecast the probability distribution of the flight’s aggregate meal demand. Due to the lack of available data from public sources, synthetic data was generated to evaluate the model developed. The model was compared against three alternative models that lacked either demand uncertainty, product substitution or both to validate the value of including these elements in the decision-making model. The comparison results indicate that the inclusion of passenger load uncertainty improves the model’s reliability to a achieve the pre-defined minimum passenger satisfaction level, whereas the inclusion of product substitution improves the model’s waste minimisation capabilities at the expense of a slightly lower reliability. To the authors’ best knowledge, no previous study has attempted to exploit the risk-pooling capabilities of a substitution model to minimise wastage resulting from surplus in-flight meals.

Conservative random walk

Recently, in [11], the “coin-turning walk” was introduced on Z. It is a non-Markovian process where the steps form a (possibly) time-inhomogeneous Markov chain. In this article, we follow up the investigation by introducing analogous processes in Zd,d≥2: at time n the direction of the process is “updated” with probability pn; otherwise the next step repeats the previous one. We study some of the fundamental properties of these walks, such as transience/recurrence and scaling limits. Our results complement previous ones in the literature about “correlated” (or “Newtonian”) and “persistent” random walks.

A Flexible Top-Down Data-Driven Stochastic Model for Synthetic Load Profiles Generation

The study of the behavior of smart distribution systems under increasingly dynamic operating conditions requires realistic and time-varying load profiles to run comprehensive and accurate simulations of power flow analysis, system state estimation and optimal control strategies. However, due to the limited availability of experimental data, synthetic load profiles with flexible duration and time resolution are often needed to this purpose. In this paper, a top-down stochastic model is proposed to generate an arbitrary amount of synthetic load profiles associated with different kinds of users exhibiting a common average daily pattern. The groups of users are identified through a preliminary Ward’s hierarchical clustering. For each cluster and each season of the year, a time-inhomogeneous Markov chain is built, and its parameters are estimated by using the available data. The states of the chain correspond to equiprobable intervals, which are then mapped to a time-varying power consumption range, depending on the statistical distribution of the load profiles at different times of the day. Such distributions are regarded as Gaussian Mixture Models (GMM). Compared with other top-down approaches reported in the scientific literature, the joint use of GMM models and time-inhomogeneous Markov chains is rather novel. Furthermore, it is flexible enough to be used in different contexts and with different temporal resolution, while keeping the number of states and the computational burden reasonable. The good agreement between synthetic and original load profiles in terms of both time series similarity and consistency of the respective probability density functions was validated by using three different data sets with different characteristics. In most cases, the median values of synthetic profiles’ mean and standard deviation differ from those of the original reference distributions by no more than ±10% both within a typical day of each season and within the population of a given cluster, although with some significant outliers.

On Periodic Scheduling and Control for Networked Systems Under Random Data Loss

This article deals with networked control systems (NCSs) whose shared networks have limited communication capacity and are prone to data losses. Our contributions are two-fold. First, we present necessary and sufficient conditions on the plant and controller dynamics and the network parameters such that there exist purely time-dependent periodic scheduling sequences under which the stability of each plant is preserved for all admissible data loss signals. Second, given a period for the scheduling sequence, we identify sufficient conditions on the plant dynamics and the network parameters such that the plants admit static state-feedback controllers that are favorable for the stability conditions presented. The main apparatus for our analysis is a switched system representation of the individual plants in an NCS whose switching signals are time-inhomogeneous Markov chains. Our stability conditions involve the existence of sets of symmetric and positive definite matrices that satisfy certain (in)equalities. We identify the existence of stabilizing periodic scheduling sequences and their corresponding favorable state-feedback controllers under ideal communication between the plants and their controllers as special cases of our results. A numerical experiment is presented to demonstrate the proposed techniques.

Stick-Breaking processes, Clumping, and Markov Chain Occupation Laws

We connect the empirical or ‘occupation’ laws of certain discrete space time-inhomogeneous Markov chains, related to simulated annealing, to a novel class of ‘stick-breaking’ processes, a ‘nonexchangeable’ generalization of the Dirichlet process used in nonparametric Bayesian statistics. To make this unexpected correspondence, we examine an intermediate ‘clumped’ structure in both the time-inhomogeneous Markov chains and the stick-breaking processes, perhaps of its own interest, which records the sequence of different states visited and the scaled proportions of time spent on them. By matching the associated intermediate structures, we identify the limits of the empirical measures of the time-inhomogeneous Markov chains as types of stick-breaking processes.

Redundancy Dependence in the Context of Competing Risks and Dynamic Degradation

Parallel redundancy is a common approach to increase network availability. As a consequence of providing redundancy, the loading of that component changes, affecting its reliability. In this article, a holistic approach combining two models is introduced to assess both instantaneous and future effects on general failure rates in the context of competing risks and dynamic degradation. Concretely, both are designed to account for the duration of abnormal loading events. First, a per-unit-degradation method is introduced to map the different loading states onto a single normal operation sojourn time variable. Second, a time-inhomogeneous Markov chain is introduced to assess the irreversible degradation following a nondetrimental abnormal loading event. The numerical illustration shows that both models are necessary to correctly assess the effects of redundancy dependence if the underlying failure distribution has a shape parameter exceeding one.

Symbolic Time Series Analysis for Anomaly Detection in Measure-Invariant Ergodic Systems

Abstract This paper presents a novel framework of symbolic time series analysis (STSA) for anomaly detection in dynamical systems. The core concept is built upon a property of measure-preserving transformation (MPT) sequence, acting on a probability space with ergodic measure, that the eigenfunctions of these transformations would be time-invariant. As a result, unlike a standard STSA that is required to generate time-homogeneous Markov chains, the proposed MPT-based STSA is allowed to have time-inhomogeneous Markov chains, where the (possibly time-varying) state transition probability matrices have time-invariant eigenvectors. Such a time-invariance facilitates analysis of the dynamical system by using short-length time series of measurements. This is particularly important in applications, where the underlying dynamics and process anomalies need fast monitoring and control actions in order to mitigate any potential structural damage and/or to avoid catastrophic failures. The MPT-based STSA has been applied for low-delay detection of fatigue damage, which is a common source of failures in mechanical structures and which is known to have uncertain dynamical characteristics. The underlying algorithm has been validated with experimental data generated from a laboratory apparatus that uses ultrasonic sensors to detect fatigue damage in polycrystalline–alloy specimens. The performance of the proposed MPT-based STSA is evaluated by comparison with those of a standard STSA and a hidden Markov model (HMM) on the same experimental data. The results consistently show superior performance of the MPT-based STSA.

Stability Estimates for Finite-Dimensional Distributions of Time-Inhomogeneous Markov Chains

This paper is devoted to the study of the stability of finite-dimensional distribution of time-inhomogeneous, discrete-time Markov chains on a general state space. The main result of the paper provides an estimate for the absolute difference of finite-dimensional distributions of a given time-inhomogeneous Markov chain and its perturbed version. By perturbation, we mean here small changes in the transition probabilities. Stability estimates are obtained using the coupling method.

Stationary Equilibria of Mean Field Games with Finite State and Action Space

Mean field games formalize dynamic games with a continuum of players and explicit interaction where the players can have heterogeneous states. As they additionally yield approximate equilibria of corresponding $N$-player games, they are of great interest for socio-economic applications. However, most techniques used for mean field games rely on assumptions that imply that for each population distribution there is a unique optimizer of the Hamiltonian. For finite action spaces, this will only hold for trivial models. Thus, the techniques used so far are not applicable. We propose a model with finite state and action space, where the dynamics are given by a time-inhomogeneous Markov chain that might depend on the current population distribution. We show existence of stationary mean field equilibria in mixed strategies under mild assumptions and propose techniques to compute all these equilibria. More precisely, our results allow -- given that the generators are irreducible -- to characterize the set of stationary mean field equilibria as the set of all fixed points of a map completely characterized by the transition rates and rewards for deterministic strategies. Additionally, we propose several partial results for the case of non-irreducible generators and we demonstrate the presented techniques on two examples.

Wiener-Hopf factorization technique for time-inhomogeneous finite Markov chains

ABSTRACT This work contributes to the theory of Wiener-Hopf type factorization for finite Markov chains. This theory originated in the seminal paper [Barlow et al., Wiener-Hopf factorization for matrices, Séminaire de Probabilités de Strasbourg 14 (1980), pp. 324–331], which treated the case of finite time-homogeneous Markov chains. Since then, several works extended the results of [Barlow et al., Wiener-Hopf factorization for matrices, Séminaire de Probabilités de Strasbourg 14 (1980), pp. 324–331] in many directions. However, all these extensions were dealing with time-homogeneous Markov case. The first work dealing with the time-inhomogeneous situation was [Bielecki et al., Wiener-Hopf factorization for time-inhomogeneous Markov chains and its application, Probab. Math. Stat. 2019, forthcoming], where Wiener-Hopf type factorization for time-inhomogeneous finite Markov chain with piecewise constant generator matrix function was derived. In the present paper we go further: we derive and study Wiener-Hopf type factorization for time-inhomogeneous finite Markov chain with the generator matrix function being a fairly general matrix valued function of time.

Local stationarity and time-inhomogeneous Markov chains

A primary motivation of this contribution is to define new locally stationary Markov models for categorical or integer-valued data. For this initial purpose, we propose a new general approach for dealing with time-inhomogeneity that extends the local stationarity notion developed in the time series literature. We also introduce a probabilistic framework which is very flexible and allows us to consider a much larger class of Markov chain models on arbitrary state spaces, including most of the locally stationary autoregressive processes studied in the literature. We consider triangular arrays of time-inhomogeneous Markov chains, defined by some families of contracting and slowly-varying Markov kernels. The finite-dimensional distribution of such Markov chains can be approximated locally with the distribution of ergodic Markov chains and some mixing properties are also available for these triangular arrays. As a consequence of our results, some classical geometrically ergodic homogeneous Markov chain models have a locally stationary version, which lays the theoretical foundations for new statistical modeling. Statistical inference of finite-state Markov chains can be based on kernel smoothing and we provide a complete and fast implementation of such models, directly usable by the practitioners. We also illustrate the theory on a real data set. A central limit theorem for Markov chains on more general state spaces is also provided and illustrated with the statistical inference in INAR models, Poisson ARCH models and binary time series models. Additional examples such as locally stationary regime-switching or SETAR models are also discussed.

A time-inhomogeneous Markov chain and its distributed solution for message dissemination in OUSNs

Opportunistic underwater sensor networks (OUSNs) are deployed for various underwater applications, such as underwater creatures tracking and tactical surveillance. The index of freshness-cost ratio is firstly introduced to describe the objectives of spatio-temporal cost minimization and message freshness preservation for the message dissemination in OUSNs. To analyze the message propagation process, a time-inhomogeneous ergodic Markov chain is constructed, and the delivery progress of each message is determined by the number of hops from the destination to the nearest message holder, which is mapped into several propagation states of the Markov chain. Then, a message dissemination method (MDM) derived from a Markov decision process is specifically designed to improve the freshness-cost ratio. In the MDM, the message holders decide how many copies should be disseminated at each time slot, according to the Markov decision process. The performance of MDM is analyzed through simulation experiments that produce preferable tradeoff results, which indicate that MDM can deliver the messages with fewer spatio-temporal cost while preserving the message freshness as much as possible.