Inhomogeneous Markov Research Articles

Beyond Time-Homogeneity for Continuous-Time Multistate Markov Models

–Multistate Markov models are a canonical parametric approach for data modeling of observed or latent stochastic processes supported on a finite state space. Continuous-time Markov processes describe data that are observed irregularly over time, as is often the case in longitudinal medical data, for example. Assuming that a continuous-time Markov process is time-homogeneous, a closed-form likelihood function can be derived from the Kolmogorov forward equations—a system of differential equations with a well-known matrix-exponential solution. Unfortunately, however, the forward equations do not admit an analytical solution for continuous-time, time- inhomogeneous Markov processes, and so researchers and practitioners often make the simplifying assumption that the process is piecewise time-homogeneous. In this article, we provide intuitions and illustrations of the potential biases for parameter estimation that may ensue in the more realistic scenario that the piecewise-homogeneous assumption is violated, and we advocate for a solution for likelihood computation in a truly time-inhomogeneous fashion. Particular focus is afforded to the context of multistate Markov models that allow for state label misclassifications, which applies more broadly to hidden Markov models (HMMs), and Bayesian computations bypass the necessity for computationally demanding numerical gradient approximations for obtaining maximum likelihood estimates (MLEs). Supplemental materials are available online.

Neural calibration of hidden inhomogeneous Markov chains: information decompression in life insurance

Markov chains play a key role in a vast number of areas, including life insurance mathematics. Standard actuarial quantities as the premium value can be interpreted as compressed, lossy information about the underlying Markov process. We introduce a method to reconstruct the underlying Markov chain given collective information of a portfolio of contracts. Our neural architecture characterizes the process in a highly explainable way by explicitly providing one-step transition probabilities. Further, we provide an intrinsic, economic model validation to inspect the quality of the information decompression. Lastly, our methodology is successfully tested for a realistic data set of German term life insurance contracts.

Anomalous diffusion in external-force-affected deterministic systems.

This study investigates the impact of external forces on the movement of particles, specifically focusing on a type of box piecewise linear map that generates normal diffusion akin to Brownian motion. Through numerical methods, the research delves into the effects of two distinct external forces: linear forces linked to the particle's current position and periodic sinusoidal forces related to time. The results uncover anomalous dynamical behavior characterized by nonlinear growth in the ensemble-averaged mean-squared displacement (EAMSD), aging, and ergodicity breaking. Notably, the diffusion pattern of particles under linear external forces resembles an Ehrenfest double urn model, with its asymptotic EAMSD coinciding with the Langevin equationunder linear potential. Meanwhile, particle movement influenced by periodic sinusoidal forces corresponds to an inhomogeneous Markov chain, with its external force amplitude and diffusion coefficient function exhibiting a "multipeak" fractal structure. The study also provides insights into the formation of this structure through the turnstiles dynamics.

Mutual aid insurance with a three-state Markov chain

ABSTRACT Mutual aid insurance is a collective type of insurance where the policyholders share the potential losses or risks that they may face. In this paper, we establish a mathematical structure for mutual aid insurance through a three-state (good, bad and death) process, which is driven by an inhomogeneous Markov chain. The objective of maximizing an individual's lifetime utility is achieved by addressing a stochastic control problem that involves both mutual aid insurance and life insurance. We obtain the explicit expressions for optimal consumption, investment strategies, and life insurance premiums by employing the corresponding Hamilton-Jacobi-Bellman equation. In the end, we carry out a numerical analysis to show the significance of mutual aid insurance and demonstrate the optimal mutual aid insurance premium.

Stochastic occupancy modeling for spaces with irregular occupancy patterns using adaptive B-Spline-based inhomogeneous Markov Chains

This paper presents a discrete time, discrete state-space in-homogeneous Markov Chains model for stochastic occupancy modeling in spaces with irregular occupancy patterns. The goal of the model is to provide accurate predictions of occupancy numbers, enabling appropriate actions to be taken for HVAC system to maintain optimal indoor environment. The proposed Markov Chain model incorporates time in-homogeneity by coupling the time-varying model parameters using a Periodic B-Spline expansion with adaptive knots, which effectively captures patterns in occupancy activity. This method optimizes the distribution of knots based on specific occupancy characteristics observed in different types of rooms. To evaluate the effectiveness of the proposed method, six months of occupancy data collected from a meeting room are utilized. A comprehensive comparison is conducted between the proposed adaptive B-Spline method and other approaches, including the counting method and uniform B-Spline method. The comparison considers both model accuracy and complexity, using metrics such as the Akaike Information Criterion and Bayesian Information Criterion. Results indicate that the proposed model achieves more accurate predictions with fewer model parameters compared to other methods. These forecasts are particularly useful in optimizing the control of HVAC systems, where accurate predictions of future occupancy numbers are essential.

Assessment on fast simulation of wind-driven pollutant dispersion around a street canyon with regime-switching Markov chain

High-density cities shorten commutes but also create proximity to air pollution sources. Sensors coupled with simulations can achieve real-time monitoring and risk assessment, and the recently emerging data-driven models provide opportunities that allow minimizing the computation burden of physical models. This study investigates the use of a time inhomogeneous Markov chain (CLF-MC) to fast simulation of wind-driven pollutant dispersion around a street canyon. The CLF-MC model is trained and tested using synthetic datasets obtained from large eddy simulations (LES). The LES model is validated using a benchmark wind tunnel test, which yields correlation coefficients above 0.95. The results show that the CLF-MC can faithfully reproduce the cumulative distribution of air pollution concentrations at the receptor site using the optimal model configuration identified in this study. For the hold-out test dataset, the release condition is a time-varying release strength, and the CLF-MC shows improved prediction results compared to the baseline models, i.e. the weighted average deviation is 15 % and 51 % smaller, respectively. In comparison to LES, the CLF-MC model is 4.8 × 106 times faster. These findings provide scientific references for fast and accurate wind-driven pollutant dispersion simulation using the probabilistic modelling approach.

Pullback measure attractors for non-autonomous stochastic reaction-diffusion equations on thin domains

This paper is concerned with pullback measure attractors of the non-autonomous stochastic reaction-diffusion equations defined in thin domains. We first prove the existence and uniqueness of pullback measure attractors for the inhomogeneous Markov process associated with the stochastic equations. We also introduce the concept of complete orbits for this sort of systems and use these special solutions to characterize the structures of pullback measure attractors. Then we establish the upper semi-continuity of these attractors as a family of thin domains collapses into a lower-dimensional domain.

Strong Ergodicity in Nonhomogeneous Markov Systems with Chronological Order

In the present, we study the problem of strong ergodicity in nonhomogeneous Markov systems. In the first basic theorem, we relax the fundamental assumption present in all studies of asymptotic behavior. That is, the assumption that the inherent inhomogeneous Markov chain converges to a homogeneous Markov chain with a regular transition probability matrix. In addition, we study the practically important problem of the rate of convergence to strong ergodicity for a nonhomogeneous Markov system (NHMS). In a second basic theorem, we provide conditions under which the rate of convergence to strong ergodicity is geometric. With these conditions, we in fact relax the basic assumption present in all previous studies, that is, that the inherent inhomogeneous Markov chain converges to a homogeneous Markov chain with a regular transition probability matrix geometrically fast. Finally, we provide an illustrative application from the area of manpower planning.

Strongly Convergent Homogeneous Approximations to Inhomogeneous Markov Jump Processes and Applications

The study of time-inhomogeneous Markov jump processes is a traditional topic within probability theory that has recently attracted substantial attention in various applications. However, their flexibility also incurs a substantial mathematical burden which is usually circumvented by using well-known generic distributional approximations or simulations. This article provides a novel approximation method that tailors the dynamics of a time-homogeneous Markov jump process to meet those of its time-inhomogeneous counterpart on an increasingly fine Poisson grid. Strong convergence of the processes in terms of the Skorokhod J1 metric is established, and convergence rates are provided. Under traditional regularity assumptions, distributional convergence is established for unconditional proxies, to the same limit. Special attention is devoted to the case where the target process has one absorbing state and the remaining ones transient, for which the absorption times also converge. Some applications are outlined, such as univariate hazard-rate density estimation, ruin probabilities, and multivariate phase-type density evaluation. Funding: M. Bladt and O. Peralta would like to acknowledge financial support from the Swiss National Science Foundation Project 200021_191984. O. Peralta acknowledges financial support from NSF Award #1653354 and AXA Research Fund Award on “Mitigating risk in the wake of the pandemic”.

Generating and understanding human daily activity sequences using Time-Varying Markov Chain models

Human activity patterns have raised broad interests across fields and activity-based modeling has been applied to analyze travel behaviors, effects of policies, etc. In this paper, we examined daily activity sequences from large volumes of datasets collected in the Tokyo Metropolitan Area over thirty years. We show that, using a simple time-varying (inhomogeneous) Markov Chain model, activity sequences at the individual level can be generated with high accuracy. The predictive power of the model is comparable to neural network models when generating activity sequences for each administrative region. The results are surprising considering the huge number of potential activity sequences and the complicated process of activity planning. A fundamental relationship between time-varying Markov Chain models and Multinomial Logit models is recognized to understand the unexpected accuracy. A new mechanism is proposed on how people make daily activity plans from the perspective of cost and utility, where the probability of one plan being chosen is the same as the probability of it being generated by the Markov Chain model. From this perspective, the Markov process becomes a utility-based model. Furthermore, given the average time-varying transition probabilities over a large spatial and temporal scale, the Markov Chain model for any specific area is reconstructed with six parameters. The generation accuracy using the reconstructed model is examined. Moreover, the statistical characteristics of the costs assigned to the Markov Chain model are investigated.

Aggregate Markov models in life insurance: estimation via the EM algorithm

In this paper, we consider statistical estimation of time–inhomogeneous aggregate Markov models. Unaggregated models, which corresponds to Markov chains, are commonly used in multi–state life insurance to model the biometric states of an insured. By aggregating microstates to each biometric state, we are able to model dependencies between transitions of the biometric states as well as the distribution of occupancy in these. This allows for non–Markovian modelling in general. Since only paths of the macrostates are observed, we develop an expectation–maximisation (EM) algorithm to obtain maximum likelihood estimates of transition intensities on the micro level. Special attention is given to a semi-Markovian case, known as the reset property, which leads to simplified estimation procedures where EM algorithms for inhomogeneous phase–type distributions can be used as building blocks. We provide a numerical example of the latter in combination with piecewise constant transition rates in a three–state disability model with data simulated from a time–inhomogeneous semi–Markov model. Comparisons of our fits with more classic GLM-based fits as well as true and empirical distributions are provided to relate our model to existing models and their tools.

Recursive Parameter Estimation and Its Convergence for Multivariate Normal Hidden Markov Inhomogeneous Models

In this paper, will discussed parameter estimation and convergence analysis of multivariate normal hidden inhomogeneous Markov models. The results of this research show that by using the expectation maximization algorithm, a sequence of parameter estimators converges to a stationary point of the likelihood function in a monotonically increasing manner.

Numerical Computation of Distributions in Finite-State Inhomogeneous Continuous Time Markov Chains, Based on Ergodicity Bounds and Piecewise Constant Approximation

In this paper it is shown, that if a possibly inhomogeneous Markov chain with continuous time and finite state space is weakly ergodic and all the entries of its intensity matrix are locally integrable, then, using available results from the perturbation theory, its time-dependent probability characteristics can be approximately obtained from another Markov chain, having piecewise constant intensities and the same state space. The approximation error (the taxicab distance between the state probability distributions) is provided. It is shown how the Cauchy operator and the state probability distribution for an arbitrary initial condition can be calculated. The findings are illustrated with the numerical examples.

Polynomial Recurrence of Time-inhomogeneous Markov Chains

This paper is devoted to establishing conditions that guarantee the existence of a p-th moment of the time it takes for a timeinhomogeneous Markov chain to hit some set C. We modified the well-known Drift Condition from the theory of homogeneous Markov chains. We demonstrated that the inhomogeneous Markov chain may be polynomially recurrent while exhibiting different dynamics from its homogeneous counterpart.

Pairwise Markov Models and Hybrid Segmentation Approach

The article studies segmentation problem (also known as classification problem) with pairwise Markov models (PMMs). A PMM is a process where the observation process and underlying state sequence form a two-dimensional Markov chain, it is a natural generalization of a hidden Markov model. To demonstrate the richness of the class of PMMs, we examine closer a few examples of rather different types of PMMs: a model for two related Markov chains, a model that allows to model an inhomogeneous Markov chain as a conditional marginal process of a homogeneous PMM, and a semi-Markov model. The segmentation problem assumes that one of the marginal processes is observed and the other one is not, the problem is to estimate the unobserved state path given the observations. The standard state path estimators often used are the so-called Viterbi path (a sequence with maximum state path probability given the observations) or the pointwise maximum a posteriori (PMAP) path (a sequence that maximizes the conditional state probability for given observations pointwise). Both these estimators have their limitations, therefore we derive formulas for calculating the so-called hybrid path estimators which interpolate between the PMAP and Viterbi path. We apply the introduced algorithms to the studied models in order to demonstrate the properties of different segmentation methods, and to illustrate large variation in behaviour of different segmentation methods in different PMMs. The studied examples show that a segmentation method should always be chosen with care by taking into account the purpose of modelling and the particular model of interest.

Statistical learning for train delays and influence of winter climate and atmospheric icing

This study investigated the climate effect under consecutive winters on the arrival delay of high-speed passenger trains. Inhomogeneous Markov chain model and stratified Cox model were adopted to account for the time-varying risks of train delays. The inhomogeneous Markov chain modelling used covariates weather variables, train operational direction, and findings from the primary delay analysis through stratified Cox model. The results showed that temperature, snow depth, ice/snow precipitation, and train operational direction, significantly impacted the arrival delay. Further, by partitioning the train line into three segments as per transition intensity, the model identified that the middle segment had the highest chance of a transfer from punctuality to delay, and the last segment had the lowest probability of recovering from delayed state. The performance of the fitted inhomogeneous Markov chain model was evaluated by the walk-forward validation method, which indicated that approximately 9% of trains may be misclassified as having arrival delays by the fitted model at a measuring point on the train line. With the model performance, the fitted model could be beneficial for both travellers to plan their trips reasonably and railway operators to design more efficient and wiser train schedules as per weather condition.

Stochastic control with inhomogeneous regime switching: Application to consumption and investment with unemployment and reemployment

This paper presents a stochastic control problem with an inhomogeneous regime switching and applies it to a consumption and investment model. We prove that the inhomogeneous Markov chain is a semimartingale, providing a basis to extend the HJB equations with regime switching to inhomogeneous cases. Explicit solutions are obtained by solving the corresponding HJB equations. Additionally, we study the impact of different levels of unemployment income and intensities of unemployment and reemployment on consumption and value function. Our findings suggest that improving unemployment security and reemployment intensity can increase overall happiness and improve the total value function.

Integrated Color and Intensity MSF Scheme for Image Retrieval Systems

Markov Stationary Features (MSF) based on homogeneous Markov chain model, for content-based image analysis, is getting popularity nowadays. It not only considers the distribution of colors, just as the histogram method does, but also characterizes the spatial co-occurrence of histogram patterns. However, handling a large-scale database of images with a degree of heterogeneity, a simple MSF method is not sufficient to discriminate the images as one requires. In this paper, an Integrated Color and Intensity MSF (ICI-MSF) based on non-homogeneous Markov Chain is proposed to overcome this shortcoming. By incorporating spatial co-occurrence of image intensities with the spatial co-occurrence information of colors and exploiting time inhomogeneous Markov chain concept, it is possible to improve certain aspects of the existing methods. Without compromising effectiveness and robustness, the method proposed in this paper keeps the feature level simplicity. Widely recognized databases namely WANG1000 and Corel10800, are used to evaluate and compare the performance of the proposed algorithm with the existing methods. The experimental results justify the effectiveness of the proposed method.

Infrastructure deterioration modeling with an inhomogeneous continuous time Markov chain: A latent state approach with analytic transition probabilities

AbstractMarkov chains have been widely used to characterize performance deterioration of infrastructure assets, to model maintenance effectiveness, and to find the optimal intervention strategies. For long‐lived assets such as bridges, the time‐homogeneity assumptions of Markov chains should be carefully checked. For this purpose, this research proposes a regime‐switching continuous‐time Markov chain of which the state transition probabilities depend on another, latent, Markov chain that characterizes the overall aging regime of an asset. With the aid of a state‐augmentation technique, closed‐form solutions for the transition probabilities are analytically derived, making the statistical analysis simple. A case study is presented using the open Ontario Bridge Condition data for provincial highway bridges. The case study demonstrates that the proposed method allows to (1) estimate a statistically superior model to the homogeneous Markov chain and (2) obtain results with comparable accuracy in approximately 48% of the computation time of the state‐of‐the‐art inhomogeneous Markov chain.

Probability of Response in the First Sixteen Weeks After Starting Biologics: An Analysis of Juvenile Idiopathic Arthritis Biologics Trials.

Most juvenile idiopathic arthritis (JIA) biologic disease-modifying antirheumatic drugs (bDMARDs) trials used an open-label run-in period followed by randomized medication withdrawal. We used data from the run-in period of 4 bDMARD trials to 1) delineate early response trajectory to bDMARDs and 2) identify predictors of early response. Data from the first 16 weeks of 4 bDMARD trials were used. The primary outcome was the American College of Rheumatology (ACR) Pediatric 50 (Pedi 50) response criteria: clinically significant response defined as ACR Pedi 50 or greater. The secondary outcome was the clinical Juvenile Arthritis Disease Activity Score in 10 joints (cJADAS10) minimal disease activity state. Response transition rates and predictors were modeled using an inhomogeneous Markov multistate model. Five hundred thirty-two participants (70% receiving methotrexate, 41% prednisone) were included. By month 4, the probability of attaining ACR Pedi 50 or greater was 0.698. If ACR Pedi 50 or more was not achieved by month 1, the probability of achieving it by month 4 was 0.60. If ACR Pedi 50 or more was not achieved by month 3, the probability of achieving this by month 4 was 0.31. Age at diagnosis, disease duration, baseline rheumatoid factor, and active joint counts predicted ACR and cJADAS state transitions, adjusted for concomitant treatment. No response ACR Pedi 50 or more by month 1 after treatment was associated with a 0.60 probability of responding by month 4, but not responding by month 3 was associated with a 0.31 probability of response by month 4. Baseline disease duration, rheumatoid factor, and active joint counts predicted early treatment response (ACR and cJADAS10 states).