Discrete-time Markov Chain Research Articles

A Discrete Model for the Evolution of Infection Prior to Symptom Onset

We consider a between-host model for a single epidemic outbreak of an infectious disease. According to the progression of the disease, hosts are classified in regard to the pathogen load. Specifically, we are assuming four phases: non-infectious asymptomatic phase, infectious asymptomatic phase (key-feature of the model where individuals show up mild or no symptoms), infectious symptomatic phase and finally an immune phase. The system takes the form of a non-linear Markov chain in discrete time where linear transitions are based on geometric (main model) or negative-binomial (enhanced model) probability distributions. The whole system is reduced to a single non-linear renewal equation. Moreover, after linearization, at least two meaningful definitions of the basic reproduction number arise: firstly as the expected secondary asymptomatic cases produced by an asymptomatic primary case, and secondly as the expected number of symptomatic individuals that a symptomatic individual will produce. We study the evolution of infection transmission before and after symptom onset. Provided that individuals can develop symptoms and die from the disease, we take disease-induced mortality as a measure of virulence and it is assumed to be positively correlated with a weighted average transmission rate. According to our findings, transmission rate of the infection is always higher in the symptomatic phase yet under a suitable condition, most of the infections take place prior to symptom onset.

Sequences of random matrices modulated by a discrete-time Markov chain*

In this work, we consider a number of matrix-valued random sequences that are modulated by a discrete-time Markov chain having a finite space. Assuming that the state space of the Markov chain is large, our main effort in this paper is devoted to reducing the computation complexity. To achieve this goal, our formulation uses time-scale separation of the Markov chain. The state-space of the Markov chain is split into subspaces. Next, the states of the Markov chain in each subspace are aggregated into a “super” state. Then we normalize the matrix-valued sequences that are modulated by the two-time-scale Markov chain. Under simple conditions, we derive a scaling limit of the centered and scaled sequence by using a martingale averaging approach. The study is carried out through a functional. It is shown that the scaled and interpolated sequence converges weakly to a switching diffusion.

Mobile agent path planning under uncertain environment using reinforcement learning and probabilistic model checking

The major challenge in mobile agent path planning, within an uncertain environment, is effectively determining an optimal control model to discover the target location as quickly as possible and evaluating the control system’s reliability. To address this challenge, we introduce a learning-verification integrated mobile agent path planning method to achieve both the effectiveness and the reliability. More specifically, we first propose a modified Q-learning algorithm (a popular reinforcement learning algorithm), called QEA−learning algorithm, to find the best Q-table in the environment. We then determine the location transition probability matrix, and establish a probability model using the assumption that the agent selects a location with a higher Q-value. Secondly, the learnt behaviour of the mobile agent based on QEA−learning algorithm, is formalized as a Discrete-time Markov Chain (DTMC) model. Thirdly, the required reliability requirements of the mobile agent control system are specified using Probabilistic Computation Tree Logic (PCTL). In addition, the DTMC model and the specified properties are taken as the input of the Probabilistic Model Checker PRISM for automatic verification. This is preformed to evaluate and verify the control system’s reliability. Finally, a case study of a mobile agent walking in a grids map is used to illustrate the proposed learning algorithm. Here we have a special focus on the modelling approach demonstrating how PRISM can be used to analyse and evaluate the reliability of the mobile agent control system learnt via the proposed algorithm. The results show that the path identified using the proposed integrated method yields the largest expected reward.

Process Mining IPTV Customer Eye Gaze Movement Using Discrete-Time Markov Chains

Human-Computer Interaction (HCI) research has extensively employed eye-tracking technologies in a variety of fields. Meanwhile, the ongoing development of Internet Protocol TV (IPTV) has significantly enriched the TV customer experience, which is of great interest to researchers across academia and industry. A previous study was carried out at the BT Ireland Innovation Centre (BTIIC), where an eye tracker was employed to record user interactions with a Video-on-Demand (VoD) application, the BT Player. This paper is a complementary and subsequent study of the analysis of eye-tracking data in our previously published introductory paper. Here, we propose a method for integrating layout information from the BT Player with mining the process of customer eye movement on the screen, thereby generating HCI and Industry-relevant insights regarding user experience. We incorporate a popular Machine Learning model, a discrete-time Markov Chain (DTMC), into our methodology, as the eye tracker records each gaze movement at a particular frequency, which is a good example of discrete-time sequences. The Markov Model is found suitable for our study, and it helps to reveal characteristics of the gaze movement as well as the user interface (UI) design on the VoD application by interpreting transition matrices, first passage time, proposed ‘most likely trajectory’ and other Markov properties of the model. Additionally, the study has revealed numerous promising areas for future research. And the code involved in this study is open access on GitHub.

Variational formulas for asymptotic variance of general discrete-time Markov chains

The asymptotic variance is an important criterion to evaluate the performance of Markov chains, especially for the central limit theorems. We give the variational formulas for the asymptotic variance of discrete-time (non-reversible) Markov chains on general state space. The variational formulas provide many applications, extending the classical Peskun’s comparison theorem to non-reversible Markov chains, and obtaining several comparison theorems between Markov chains with various perturbations.

이산 시각 대기 행렬 모형의 정보 신선도

The age of information (AoI) was proposed to quantify the freshness of information about the status of a remote source system, which is defined as the amount of time that has elapsed since a packet was created at its source. This paper analyzes the age of information of a discrete time Geo/D/1/1 status update system. For this purpose, the system is modeled as a discrete-time two-state Markov chain. The stationary probability distributions for peak AoI and AoI are obtained. The average peak AoI, the average AoI, and the freshness ratio of information are also derived. Some numerical results of the analysis are presented.

Tumor growth prediction and classification based on the KNN algorithm and discrete-time Markov chains (DTMC)

Tumor growth prediction and classification based on the KNN algorithm and discrete-time Markov chains (DTMC)

MSI-A: An Energy Efficient Approximated Cache Coherence Protocol

Energy consumption has become an essential factor in designing modern computer system architecture. Because of physical limits, the termination of Moore’s law and Dennard’s scaling has forced the computer design community to investigate new approaches to meet the requirements for computing resources. Approximate computing has emerged as a promising method for reducing energy consumption while trading a controllable quality loss. This paper asserts that an approximated cache coherence protocol preserves overall energy for computation. We can approximate the cache coherence protocol by adding approximated cache lines to a certain level without hindering the output. This paper introduces an enhanced approximated version of the MSI (Modified Shared Invalid) cache coherence protocol MSI-A (Modified Shared Invalid-Approx). We have verified MSI-A and MSI by employing LTL specifications in the NuSMV model checker. To illustrate the benefits of MSI-A, we have added DTMC (Discrete-Time Markov Chain) with PCTL (Probabilistic Computational Tree Logic). Although the PCTL proves the theory of approximation, we have also simulated the MSI-A in the TEJAS hardware simulator on PARSEC 3.0 to investigate the energy gains and cycle gains of MSI-A in varied applications. The cache lines considered to be approx are between 10 and 30 percent. Each application benefited from approximation according to its nature, and VIPS has indicated a total energy gain of 30.18 percent.

Opportunistic Routing Aided Cooperative Communication MRC Network With Energy-Harvesting Nodes

In this research, we present a cooperative communication network based on two energy-harvesting (EH) decode-and-forward (DF) relays that contain the harvest-store-use (HSU) architecture and may harvest energy from the surrounding environment by utilizing energy buffers. To improve the performance of this network, an opportunistic routing (OR) algorithm is presented that takes into account channel status information, relay position, and energy buffer status, as well as using the maximal ratio combining (MRC) at the destination to combine the signals received from the source and relays. The theoretical expressions for limiting distribution of energy stored in infinite-size buffers are derived from the discrete-time continuous-state space Markov chain model (DCSMC). In addition, the theoretical expressions for outage probability, throughput, and per-packet time slot cost in the network are obtained utilizing both the limiting distributions of energy buffers and the probabilities of transmitter candidates set. Through numerous simulation results, it is demonstrated that simulation results match with corresponding theoretical results.

Task-Load-Aware Game-Theoretic Framework for Wireless Federated Learning

Federated learning (FL) can protect data privacy but has difficulties in motivating user equipment (UE) to engage in task training. This letter proposes a Bertrand-game based framework to address the incentive problem, where a model owner (MO) issues an FL task and the employed UEs help train the model by using their local data. Specially, we consider the impact of time-varying task load and channel quality on UE’s motivation to engage in the FL task. We adopt the finite-state discrete-time Markov chain (FSDT-MC) to predict these parameters during the FL task. Depending on the performance metrics set by the MO and the estimated energy cost of the FL task, each UE seeks to maximize its profit. We obtain the Nash equilibrium (NE) of the game in closed form, and develop a distributed iterative algorithm to find it. Finally, the simulation result verifies the effectiveness of the proposed approach.

Modeling the Dynamics of the Random Demand Inventory Management System

At any given time, a product stock manager is expected to carry out activities to check his or her holdings in general and to monitor the condition of the stock in particular. He should monitor the level or quantity available of a given product, of any item. On the basis of the observation made in relation to the movements of previous periods, he may decide to order or not a certain quantity of products. This paper discusses the applicability of discrete-time Markov chains in making relevant decisions for the management of a stock of COTRA-Honey products. A Markov chain model based on the transition matrix and equilibrium probabilities was developed to help managers predict the likely state of the stock in order to anticipate procurement decisions in the short, medium or long term. The objective of any manager is to ensure efficient management by limiting overstocking, minimising the risk of stock-outs as much as possible and maximising profits. The determined Markov chain model allows the manager to predict whether or not to order for the period following the current period, and if so, how much.

A Markov chain-based IoT system for monitoring and analysis of urban air quality.

Severe deterioration of urban air quality in Asian cities is the cause of a large number of deaths every year. A Markov chain-based IoT system is developed in this study to monitor, analyze, and predict urban air quality. The proposed sensing setup is integrated with an automobile and is used for collecting air quality information. An Android application is used to transfer and store the sensed data in the data cloud. The data stored is used to generate the transition matrix of the AQI states and calculate return periods for each AQI state. The estimated time interval after which an AQI event recurs or is repeated is known as return period. The actual return periods for each AQI state at the test locations in Delhi-NCR are compared with those predicted using discrete time Markov chain (DTMC) models. Average absolute forecast error using our model was found to be 3.38% and 4.06%, respectively, at the selected locations.

Discrete-time zero-sum games for Markov chains with risk-sensitive average cost criterion

We study zero-sum stochastic games for controlled discrete time Markov chains with risk-sensitive average cost criterion with countable/compact state space and Borel action spaces. The payoff function is nonnegative and possibly unbounded for countable state space case and for compact state space case it is a real-valued and bounded function. For countable state space case, under a certain Lyapunov type stability assumption on the dynamics we establish the existence of the value and a saddle point equilibrium. For compact state space case we establish these results without any Lyapunov type stability assumptions. Using the stochastic representation of the principal eigenfunction of the associated optimality equation, we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. Also, we present and analyze an illustrative example.

Use of Markov Chain Simulation in Long Term Care Insurance

The aim of this paper is to present the use of simulations of non-homogeneous Markov chains in discrete time in the context of the problem of long-term care delivery. The object of investigation is to model the distribution of clients into different states during specified time steps, then to estimate the average time a client stays in a given state, as well as to estimate the insurance premiums. Within the use of the Monte Carlo simulation method, the focus is on providing approaches that ensure more accurate results in the context of the number of simulations performed. Based on the statistical processing of the data obtained from the simulations, it is possible to obtain the information necessary for the provision of resources for the provision of health care and for the determination of the aforementioned premiums. For the implementation of the above techniques and their graphical presentation available packages such as markovchain, ggplot2 or custom code created using the R language were used.

A Markov Chain Approach to the Pattern of Blood Donation Status at a Blood Service Centre in Zimbabwe

Background: Blood donors’ behaviour towards blood donation is not easily predictable and can be considered a stochastic random variable. A four-state Markov chain technique was defined and adopted in this study. The transition probabilities of blood donation within the four identified states, viz: new, regular, occasional, and lapsed donors were used to making further inferences about the dynamics in blood donation in Harare, Zimbabwe. Objectives: The paper presents a four-state Discrete Time Markov Chain (DTMC) model in analysing the changes in blood donation status over the four-year study period. Methodology: A transition probabilities matrix was developed and parameters estimated using the maximum likelihood method and two other approaches, and inferences were made based on the resultant transition matrix. Results: About 56% of new donors made at least one repeat donation and became regular donors within the first year, and the numbers gradually declined with time, whilst the lapsed donors increased from 35.6% in the second year to 55.6% in year 4. The long-run probabilities tell the same, with 80.9% of blood donations becoming lapsed in the long run. Depending on the current state of donation, new or regular donations will likely move to the regular donation state in the following time step (year). On the other end, occasional and lapsed donations have a higher probability of entering the lapsed donation state in the following time step (year). Conclusion: The paper provides useful insights into the Markovian transition probabilities among the blood donation states, and this has implications on future blood donors’ pool and blood bank inventory in Zimbabwe. The decline in the number of donors who make repeat donations is a worrisome trend since regular donations are the lifeline of any blood service centre.

An optimal ordering inventory policy when the purchase price follows a discrete-time Markov chain

Most of the time retailers are offered discount prices that are occurring at random points in time. One such scenario is the supplier offering discounts to the retailers to increase market share, cash flow, and to reduce the inventory of specific items. Surprisingly no models exist in the literature survey to model the randomly occurring discount price under discrete time. The primary objective of this article is to develop an optimal inventory policy for a retailer when the discount price arrives randomly and the time variable is discrete. Under such a scenario, the optimal inventory policy can be obtained by considering the variations in the purchase price as a discrete-time Markov chain. The renewal reward theorem is used to derive the expected profit per unit of time. To illustrate the application of the developed model a real case study is considered. The optimal solution of the developed model is compared with the EOQ policy and found that the developed model solution provides better profit. This justifies the significance of the developed model. The inventory manager can use the developed model as a tool to obtain the optimal solution for any two price problems that repeat randomly.

Distorted probability operator for dynamic portfolio optimization in times of socio-economic crisis

A robust optimal control of discrete time Markov chains with finite terminal T and bounded costs or wealth using probability distortion is studied. The time inconsistency of these distortion operators and hence its lack of dynamic programming are discussed. Due to that, dynamic versions of these operators are introduced, and its availability for dynamic programming is demonstrated. Based on dynamic programming algorithm, existence of the optimal policy is justified and an application of the theory to portfolio optimization along with a numerical study is also presented.

The multilevel hierarchical data EM-algorithm. Applications to discrete-time Markov chain epidemic models

The theory of multilevel hierarchical data Expectation Maximization (EM)-algorithm is introduced via discrete time Markov chain (DTMC) epidemic models. A general model for a multilevel hierarchical discrete data is derived. The observed sample Y in the system is a stochastic incomplete data, and the missing data Z exhibits a multilevel hierarchical data structure. The EM-algorithm to find ML-estimates for parameters in the stochastic system is derived. Applications of the EM-algorithm are exhibited in the two DTMC models, to find ML-estimates of the system parameters. Numerical results are given for influenza epidemics in the state of Georgia (GA), USA.

A stable sensory map emerges from a dynamic equilibrium of neurons with unstable tuning properties.

Recent long-term measurements of neuronal activity have revealed that, despite stability in large-scale topographic maps, the tuning properties of individual cortical neurons can undergo substantial reformatting over days. To shed light on this apparent contradiction, we captured the sound response dynamics of auditory cortical neurons using repeated 2-photon calcium imaging in awake mice. We measured sound-evoked responses to a set of pure tone and complex sound stimuli in more than 20,000 auditory cortex neurons over several days. We found that a substantial fraction of neurons dropped in and out of the population response. We modeled these dynamics as a simple discrete-time Markov chain, capturing the continuous changes in responsiveness observed during stable behavioral and environmental conditions. Although only a minority of neurons were driven by the sound stimuli at a given time point, the model predicts that most cells would at least transiently become responsive within 100days. We observe that, despite single-neuron volatility, the population-level representation of sound frequency was stably maintained, demonstrating the dynamic equilibrium underlying the tonotopic map. Our results show that sensory maps are maintained by shifting subpopulations of neurons "sharing" the job of creating a sensory representation.

Entropy rate of random walks on complex networks under stochastic resetting.

Stochastic processes under resetting at random times have attracted a lot of attention in recent years and served as illustrations of nontrivial and interesting static and dynamic features of stochastic dynamics. In this paper, we aim to address how the entropy rate is affected by stochastic resetting in discrete-time Markovian processes, and we explore nontrivial effects of the resetting in the mixing properties of a stochastic process. In particular, we consider resetting random walks (RRWs) with a single resetting node on three different types of networks: degree-regular random networks, a finite-size Cayley tree, and a Barabási-Albert (BA) scale-free network, and we compute the entropy rate as a function of the resetting probability γ. Interestingly, for the first two types of networks, the entropy rate shows a nonmonotonic dependence on γ. There exists an optimal value of γ at which the entropy rate reaches a maximum. Such a maximum is larger than that of maximal-entropy random walks (MREWs) and standard random walks (SRWs) on the same topology, while for the BA network the entropy rate of RRWs either shows a unique maximum or decreases monotonically with γ, depending upon the choice of the resetting node. When the maximum entropy rate of RRWs exists, it can be higher or lower than that of MREWs or SRWs. Our study reveals a nontrivial effect of stochastic resetting on nonequilibrium statistical physics.