Finite-state Discrete-time Markov Chain Research Articles

Task-Load-Aware Game-Theoretic Framework for Wireless Federated Learning

Federated learning (FL) can protect data privacy but has difficulties in motivating user equipment (UE) to engage in task training. This letter proposes a Bertrand-game based framework to address the incentive problem, where a model owner (MO) issues an FL task and the employed UEs help train the model by using their local data. Specially, we consider the impact of time-varying task load and channel quality on UE’s motivation to engage in the FL task. We adopt the finite-state discrete-time Markov chain (FSDT-MC) to predict these parameters during the FL task. Depending on the performance metrics set by the MO and the estimated energy cost of the FL task, each UE seeks to maximize its profit. We obtain the Nash equilibrium (NE) of the game in closed form, and develop a distributed iterative algorithm to find it. Finally, the simulation result verifies the effectiveness of the proposed approach.

Coupling compartmental models with Markov chains and measure evolution equations to capture virus mutability

The COVID-19 pandemic lit a fire under researchers who have subsequently raced to build models which capture various physical aspects of both the biology of the virus and its mobility throughout the human population. These models could include characteristics such as different genders, ages, frequency of interactions, mutation of virus, etc. Here, we propose two mathematical formulations to include virus mutation dynamics. The first uses a compartmental epidemiological model coupled with a discrete-time finite-state Markov chain. If one includes a nonlinear dependence of the transition matrix on current infected, the model is able to reproduce pandemic waves due to different variants. The second approach expands such an idea to a continuous state-space leveraging a combination of ordinary differential equations with an evolution equation for measure. This approach allows to include reinfections with partial immunity with respect to variants genetically similar to that of first infection.

Detecting State Transitions of a Markov Source: Sampling Frequency and Age Trade-off

We consider a finite-state Discrete-Time Markov Chain (DTMC) source that can be sampled for detecting the events when the DTMC transits to a new state. Our goal is to study the trade-off between sampling frequency and staleness in detecting the events. We argue that, for the problem at hand, using Age of Information (AoI) for quantifying the staleness of a sample is conservative and therefore, study another freshness metric <i>age penalty</i>, which is defined as the time elapsed since the first transition out of the most recently observed state. We study two optimization problems: minimize average age penalty subject to an average sampling frequency constraint, and minimize average sampling frequency subject to an average age penalty constraint; both are Constrained Markov Decision Problems. We solve them using the Lagrangian MDP approach, where we also provide structural results that reduce the search space. Our numerical results demonstrate that the computed Markov policies not only outperform optimal periodic sampling policies, but also achieve sampling frequencies close to or lower than that of an optimal clairvoyant (non-causal) sampling policy, if a small age penalty is allowed.

A stochastic model of a production-inventory system with consideration of production disruption

In this paper, we develop a stochastic production-inventory model that faces random production disruption. In each fixed time planning horizon, the inventory position is affected by production disruption. The production disruption time and the disruption recovery time in a single planning horizon are stochastic. The process of the inventory levels at the beginning of the time horizon is formulated by using a finite state discrete-time Markov chain. We derive expressions of the transition probability and the long-run average cost. The optimum solution is obtained through numerical experiments. The result shows that under the circumstance of production time constraint, the incorporation of safety stock in a disruption prone production-inventory system helps to minimise the long run average cost. Finally, sensitivity analysis has been given to demonstrate the usefulness of the model. This model assists decision makers to not only determine the optimal safety stock quantity but also reveal distinct effect and/or joint impacts of individual cost parameters, disruption probability on the decision regarding selection of safety stock.

A stochastic model of a production-inventory system with consideration of production disruption

In this paper, we develop a stochastic production-inventory model that faces random production disruption. In each fixed time planning horizon, the inventory position is affected by production disruption. The production disruption time and the disruption recovery time in a single planning horizon are stochastic. The process of the inventory levels at the beginning of the time horizon is formulated by using a finite state discrete-time Markov chain. We derive expressions of the transition probability and the long-run average cost. The optimum solution is obtained through numerical experiments. The result shows that under the circumstance of production time constraint, the incorporation of safety stock in a disruption prone production-inventory system helps to minimise the long run average cost. Finally, sensitivity analysis has been given to demonstrate the usefulness of the model. This model assists decision makers to not only determine the optimal safety stock quantity but also reveal distinct effect and/or joint impacts of individual cost parameters, disruption probability on the decision regarding selection of safety stock.

Spatial Throughput Analysis and Transmission Strategy Design in Energy Harvesting Cognitive Radio Networks

In this paper, we consider an energy harvesting cognitive radio network where each secondary transmitter (ST) harvests radio frequency energy from ambient primary transmitters (PTs), and communicates with its secondary receiver (SR) which suffers co-channel interference from PTs. A positive correlation is observed between the harvested energy at the ST and the aggregate interference at the SR, which illustrates that an ST will have a higher probability to harvest enough energy if there is strong interference at the SR. To exploit the positive correlation, we propose an interference threshold-based transmission strategy for STs, so as to protect secondary transmissions as well as increase the amount of harvested energy. We model the battery level of each ST as a finite state discrete-time Markov chain and derive the expression of the secondary spatial throughput as a function of the transmission probability and coverage probability. We investigate the impact of interference threshold and density of PTs on the secondary spatial throughput, and provide guidelines in the optimal design of these two factors to maximize the spatial throughput. To further improve the spatial throughput of a secondary network, we consider the successive interference cancellation strategy at SRs, and reveal its superiority in the high density regime of PTs.

Validating User Flows to Protect Software Defined Network Environments

Software Defined Network is a promising network paradigm which has led to several security threats in SDN applications that involve user flows, switches, and controllers in the network. Threats as spoofing, tampering, information disclosure, Denial of Service, flow table overloading, and so on have been addressed by many researchers. In this paper, we present novel SDN design to solve three security threats: flow table overloading is solved by constructing a star topology-based architecture, unsupervised hashing method mitigates link spoofing attack, and fuzzy classifier combined with L1-ELM running on a neural network for isolating anomaly packets from normal packets. For effective flow migration Discrete-Time Finite-State Markov Chain model is applied. Extensive simulations using OMNeT++ demonstrate the performance of our proposed approach, which is better at preserving holding time than are other state-of-the-art works from the literature.

State estimation for networked systems with Markovian communication constraints and multiple packet dropouts

In this paper, the state estimation problem for discrete-time networked systems with communication constraints and random packet dropouts is considered. The communication constraint is that, at each sampling instant, there is at most one of the various transmission nodes in the networked systems is allowed to access a shared communication channel, and then the received data are transmitted to a remote estimator to perform the estimation task. The channel accessing process of those transmission nodes is determined by a finite-state discrete-time Markov chain, and random packet dropouts in remote data transmission are modeled by a Bernoulli distributed white sequence. Using Bayes’ rule and some results developed in this study, two state estimation algorithms are proposed in the sense of minimum mean-square error. The first algorithm is optimal, which can exactly compute the minimum mean-square error estimate of system state. The second algorithm is a suboptimal algorithm obtained under a lot of Gaussian hypotheses. The proposed suboptimal algorithm is recursive and has time-independent complexity. Computer simulations are carried out to illustrate the performance of the proposed algorithms.

Modeling and Analysis of Link Duration in Vehicular Ad Hoc Networks Under Different Fading Channel Conditions

Modeling and analysis of communication link duration in vehicular ad hoc networks (VANETs) is crucial as it directly affects many performance metrics such as packet delivery ratio, throughput and end-to-end delay. In this paper, we investigate the properties of communication link duration in one-dimensional VANETs for different fading channel conditions. We employ a stochastic microscopic mobility model that takes into account time dependence of the inter-vehicle distance. A discrete-time finite-state Markov chain with state dependent transition probabilities is used to model the inter-vehicle distance. The presence of fading channel conditions will introduce fluctuations in the received signal power. Even when the inter-vehicle distance between a given pair of vehicles is less than their transmission range, poor channel conditions may result in link failure. A modified distance transition probability matrix is derived to describe the combined effects of relative velocity among a pair of vehicles, link failure due to outage, and inter-vehicle distance. The probability distribution of the communication link duration is derived from the modified distance transition probability matrix. The analytical results obtained from the model are validated by extensive simulation results.

Model Predictive Control for Discrete-Time Systems with Random Delay and Randomly Occurring Nonlinearity

In this paper, the model predictive control problem is investigated for a class of discrete-time systems with random delay and randomly occurring nonlinearity. The randomly occurring nonlinearity, which describes the phenomena of a class of nonlinear disturbances occurring in a random way, is modeled according to a Bernoulli distributed white sequence with a known conditional probability. Moreover, the random delay is governed by a discrete-time finite-state Markov chain. The approach of delay fractioning is applied to the controller synthesis. It is shown that the proposed model predictive controller guarantees the stochastic stability of the closed-loop system. Finally, a numerical simulation is given to illustrate the effectiveness of the proposed method.

Stochastic Analysis of a Single-Hop Communication Link in Vehicular Ad Hoc Networks

A vehicular ad hoc network (VANET) is a promising addition to our future intelligent transportation systems, which supports various safety and infotainment applications. The high node mobility and frequent topology changes in VANETs impose new challenges in maintaining a long-lasting connection between network nodes. As a result, the lifetime of communication links is a crucial issue in VANET development and operation. This paper presents a probabilistic analysis of the communication link in VANETs for three vehicle density ranges. First, we present the stationary distribution of the communication link length using mesoscopic mobility models. Second, we propose a stochastic microscopic mobility model that captures time variations of intervehicle distances (distance headways). A discrete-time finite-state Markov chain with state-dependent transition probabilities is proposed to model the distance headway. Third, the proposed stochastic microscopic model and first passage time analysis are used to derive the probability distribution of the communication link lifetime. Numerical results are presented to evaluate the proposed model, which demonstrate a close agreement between analytical and simulation results.

Finite-state discrete-time Markov chain models of gene regulatory networks

In this study, Markov chain models of gene regulatory networks (GRN) are developed. These models make it possible to apply the well-known theory and tools of Markov chains to GRN analysis. A new kind of finite interaction graph called a combinatorial net is introduced to represent formally a GRN and its transition graphs constructed from interaction graphs. The system dynamics are defined as a random walk on the transition graph, which is a Markov chain. A novel concurrent updating scheme (evolution rule) is developed to determine transitions in a transition graph. The proposed scheme is based on the firing of a random set of non-steady-state vertices in a combinatorial net. It is demonstrated that this novel scheme represents an advance in asynchronicity modeling. The theorem that combinatorial nets with this updating scheme can asynchronously compute a maximal independent set of graphs is also proved. As proof of concept, a number of simple combinatorial models are presented here: a discrete auto-regression model, a bistableswitch, an Elowitz repressilator, and a self-activation model, and it is shown that these models exhibit well-known properties.

Probability-based constrained MPC for structured uncertain systems with state and random input delays

This article is concerned with probability-based constrained model predictive control (MPC) for systems with both structured uncertainties and time delays, where a random input delay and multiple fixed state delays are included. The process of input delay is governed by a discrete-time finite-state Markov chain. By invoking an appropriate augmented state, the system is transformed into a standard structured uncertain time-delay Markov jump linear system (MJLS). For the resulting system, a multi-step feedback control law is utilised to minimise an upper bound on the expected value of performance objective. The proposed design has been proved to stabilise the closed-loop system in the mean square sense and to guarantee constraints on control inputs and system states. Finally, a numerical example is given to illustrate the proposed results.

Formal Reasoning About Finite-State Discrete-Time Markov Chains in HOL

Markov chains are extensively used in modeling different aspects of engineering and scientific systems, such as performance of algorithms and reliability of systems. Different techniques have been developed for analyzing Markovian models, for example, Markov Chain Monte Carlo based simulation, Markov Analyzer, and more recently probabilistic model-checking. However, these techniques either do not guarantee accurate analysis or are not scalable. Higher-order-logic theorem proving is a formal method that has the ability to overcome the above mentioned limitations. However, it is not mature enough to handle all sorts of Markovian models. In this paper, we propose a formalization of Discrete-Time Markov Chain (DTMC) that facilitates formal reasoning about time-homogeneous finite-state discrete-time Markov chain. In particular, we provide a formal verification on some of its important properties, such as joint probabilities, Chapman-Kolmogorov equation, reversibility property, using higher-order logic. To demonstrate the usefulness of our work, we analyze two applications: a simplified binary communication channel and the Automatic Mail Quality Measurement protocol.

On the Queue Dynamics of Multiuser Multichannel Cognitive Radio Networks

For protocol and system design in cognitive radio networks (CRNs), it is essential to identify which system settings or environmental conditions have great impacts on the quality-of-service (QoS) performance for secondary users (SUs) and how they influence it, which are still open issues. In this paper, an analytical framework to quantify the queue dynamics of a multi-SU multichannel CRN is developed. In the analytical framework, we consider the important lower-layer mechanisms and settings, including spectrum sensing errors, medium-access control (MAC) protocols, link adaptation technologies such as adaptive modulation and coding (AMC) and automatic repeat request (ARQ), and limited buffer size. By modeling the queue dynamics as a discrete-time finite-state Markov chain (FSMC), we derive the analytical expressions of the average queue length, packet-dropping rate, and packet-collision rate. Based on these expressions, the QoS metrics including average queueing delay, packet-loss rate, and effective throughput are obtained. Simulation and numerical results are presented to verify the accuracy of the proposed analytical framework and investigate the QoS performance of SUs.

Online Learning of Rested and Restless Bandits

In this paper we study the online learning problem involving rested and restless multiarmed bandits with multiple plays. The system consists of a single player/user and a set of K finite-state discrete-time Markov chains (arms) with unknown state spaces and statistics. At each time step the player can play M arms. The objective of the user is to decide for each step which M of the K arms to play over a sequence of trials so as to maximize its long term reward. The restless multiarmed bandit is particularly relevant to the application of opportunistic spectrum access (OSA), where a (secondary) user has access to a set of K channels, each of time-varying condition as a result of random fading and/or certain primary users' activities.

Large Deviation Bounds for Functionals of Viterbi Paths

In a number of applications, the underlying stochastic process is modeled as a finite-state discrete-time Markov chain that cannot be observed directly and is represented by an auxiliary process. The maximum a posteriori (MAP) estimator is widely used to estimate states of this hidden Markov model through available observations. The MAP path estimator based on a finite number of observations is calculated by the Viterbi algorithm, and is often referred to as the Viterbi path. It was recently shown in, and, (see also and) that under mild conditions, the sequence of estimators of a given state converges almost surely to a limiting regenerative process as the number of observations approaches infinity. This in particular implies a law of large numbers for some functionals of hidden states and finite Viterbi paths. The aim of this paper is to provide the corresponding large deviation estimates.

A self-tuning model for inflation rate dynamics

A regime-switching model is proposed to capture the structural changes in inflation dynamics. We apply a special version of the EM algorithm to find optimal parameter estimates of the model within the framework of a discrete-time finite state Markov chain that governs the switching of regimes from one state to another. The model is implemented to Canada’s consumer price index (CPI) data series and its performance is assessed by comparing its one-step ahead predictions with the actual data. We found that, within the dataset studied, a two-state Markov-switching model is sufficient to capture the dynamics of Canadian CPI series. The model being proposed is adaptive as parameters are updated upon the arrival of a new set of information. A description of how to calculate the standard errors for parameter estimates using the Fisher information matrix is provided. We also determine the optimal number of states for the Markov chain within the dataset considered via the AIC analysis.

Adaptive Importance Sampling Technique for Markov Chains Using Stochastic Approximation

For a discrete-time finite-state Markov chain, we develop an adaptive importance sampling scheme to estimate the expected total cost before hitting a set of terminal states. This scheme updates the change of measure at every transition using constant or decreasing step-size stochastic approximation. The updates are shown to concentrate asymptotically in a neighborhood of the desired zero-variance estimator. Through simulation experiments on simple Markovian queues, we observe that the proposed technique performs very well in estimating performance measures related to rare events associated with queue lengths exceeding prescribed thresholds. We include performance comparisons of the proposed algorithm with existing adaptive importance sampling algorithms on some examples. We also discuss the extension of the technique to estimate the infinite horizon expected discounted cost and the expected average cost.

Balanced importance resampling for Markov chains

In this paper, given a sequence of realizations of a discrete-time finite state Markov chain, we estimate both transition probabilities and invariant distributions using a bootstrap method. In its original form this technique requires independent and identically distributed samples. Therefore, since the sequence is “Markov”, it has to be adapted in order to fit into this framework. Block bootstrap is introduced to make this adaptation. In order to reduce computer time, balanced importance resampling is proposed, so that in the bootstrap procedure, resampling is not done uniformly , this distribution is modified in order to get variance reduction. Efficiency properties of this alternative distribution are shown, together with numerical data. A central limit theorem for multinomial random sums is also proved.