Discrete-time Markov Chain Research Articles

Multi-Dimensional Markov Chains of M/G/1 Type

We consider an irreducible discrete-time Markov process with states represented as (k, i) where k is an M-dimensional vector with non-negative integer entries, and i indicates the state (phase) of the external environment. The number n of phases may be either finite or infinite. One-step transitions of the process from a state (k, i) are limited to states (n, j) such that n ≥ k−1, where 1 represents the vector of all 1s. We assume that for a vector k ≥ 1, the one-step transition probability from a state (k, i) to a state (n, j) may depend on i, j, and n − k, but not on the specific values of k and n. This process can be classified as a Markov chain of M/G/1 type, where the minimum entry of the vector n defines the level of a state (n, j). It is shown that the first passage distribution matrix of such a process, also known as the matrix G, can be expressed through a family of nonnegative square matrices of order n, which is a solution to a system of nonlinear matrix equations.

A variance reduction technique for Monte Carlo simulations of electrons and ions in electric and magnetic fields

A computationally efficient variance reduction technique for Monte Carlo simulations of electrons and ions in weakly ionized gases is proposed. The transport of charged particles under electric and magnetic fields is expressed as a discrete-time Markov process in a grid. This results in a significant reduction of the computational time and statistical fluctuations of the computed velocity distribution functions (VDFs). The results are presented for a model gas and different values of the Hall parameter. The method is then applied to simulations of electrons in D2 and H+ ions in H2 using state-of-the-art cross sections and different values of externally applied electric and magnetic fields. It is shown that this approach allows one to study the combined effects of electric and magnetic fields on charged particles transport in a notably simple way, without employing a spherical harmonic expansion of the VDF.

MODELLING STUDENTS’ ACADEMIC PERFORMANCE AND PROGRESS: A DISCRETE-TIME MARKOV CHAIN APPROACH

Predicting students' performance has become increasingly challenging due to the large volume of data in educational databases. Academic achievement reflects learning effectiveness and serves as a key indicator of teaching quality, institutional standards, and overall student development. Higher education systems operate hierarchically, with students progressing through academic levels annually or exiting as graduates or dropouts. Understanding and evaluating student progression is vital amidst evolving educational dynamics. This study models students’ academic performance and progression using a discrete-time Markov chain approach to predict future outcomes. Data on students’ enrollment and performance for five(5) sessions were collected from the Department of Statistics, Federal University of Technology, Minna. The Markov chain model was constructed for different academic levels and their absorbing states. Key metrics, including expected time spent at each level, absorption probabilities, and graduation or withdrawal likelihoods, were estimated. The findings show that 100-level entrants have an 80.5% chance of graduating and a 19.5% risk of withdrawal, with graduation likelihoods increasing with progression—reaching 99.4% at 500-level. The forecasts from the constructed Markov chain models showed that 100-level entrants are 99.4% likely to graduate after five sessions, 200-level entrants after three sessions, and 300-level entrants after one session. The study shows that while attrition rates are higher in the early stages, students advancing beyond the 200-level exhibit strong prospects for completion. These findings underscore the university’s effective programs and support systems, particularly in retaining and advancing students beyond the critical early stages.

Joint Sampling and Transmission Policies for Minimizing Cost Under Age of Information Constraints.

In this work, we consider the problem of jointly minimizing the average cost of sampling and transmitting status updates by users over a wireless channel subject to average Age of Information (AoI) constraints. Errors in the transmission may occur and a policy has to decide if the users sample a new packet or attempt to retransmission the packet sampled previously. The cost consists of both sampling and transmission costs. The sampling of a new packet after a failure imposes an additional cost on the system. We formulate a stochastic optimization problem with the average cost in the objective under average AoI constraints. To solve this problem, we propose three scheduling policies: (a) a dynamic policy, which is centralized and requires full knowledge of the state of the system and (b) two stationary randomized policies that require no knowledge of the state of the system. We utilize tools from Lyapunov optimization theory and Discrete-Time Markov Chain (DTMC) to provide the dynamic policy and the randomized ones, respectively. Simulation results show the importance of providing the option to transmit an old packet in order to minimize the total average cost.

Performance Analysis of Wireless Sensor Networks Using Damped Oscillation Functions for the Packet Transmission Probability

Wireless sensor networks are composed of many nodes distributed in a region of interest to monitor different environments and physical variables. In many cases, access to nodes is not easy or feasible. As such, the system lifetime is a primary design parameter to consider in the design of these networks. In this regard, for some applications, it is preferable to extend the system lifetime by actively reducing the number of packet transmissions and, thus, the number of reports. The system administrator can be aware of such reporting reduction to distinguish this final phase from a malfunction of the system or even an attack. Given this, we explore different mathematical functions that drastically reduce the number of packet transmissions when the residual energy in the system is low but still allow for an adequate number of transmissions. Indeed, in previous works, where the negative exponential distribution is used, the system reaches the point of zero transmissions extremely fast. Hence, we propose different dampening functions with different decreasing rates that present oscillations to allow for packet transmissions even at the end of the system lifetime. We compare the system performance under these mathematical functions, which, to the best of our knowledge, have never been used before, to find the most adequate transmission scheme for packet transmissions and system lifetime. We develop an analytical model based on a discrete-time Markov chain to show that a moderately decreasing function provides the best results. We also develop a discrete event simulator to validate the analytical results.

Embedding and elimination for performance analysis in stochastic process algebra dtsdPBC

ABSTRACT dtsdPBC extends the well-known algebra of parallel processes, Petri box calculus (PBC), by incorporating discrete time stochastic and deterministic delays. To analyze performance in this extended calculus, the underlying semi-Markov chains, and the related (complete) and reduced discrete time Markov chains of the process expressions are built. The semi-Markov chains are extracted using the embedding method, which constructs the embedded discrete time Markov chains and calculates the sojourn time distributions in the states. The reductions of the discrete time Markov chains are obtained through the elimination method, which removes the vanishing states (those with zero sojourn times) and recalculates the transition probabilities among the tangible states (those with positive sojourn times). We prove that the reduced semi-Markov chain coincides with the reduced discrete time Markov chain, by demonstrating that an additional embedding into the reduced semi-Markov chain is needed for the reduced embedded discrete time Markov chain to match the embedded reduced discrete time Markov chain, and by comparing the respective sojourn times.

Stability and stabilization of discrete-time linear compartmental switched systems via Markov chains

The stabilizing switching signal design of discrete-time linear compartmental switched systems (DT-LCSSs) has been heretofore unsolved. It has been proven that a DT-LCSS is stabilizable if and only if it is stabilizable by a periodic switching signal. However, it still needs to be determined whether the period of a stabilizing switching signal can be confined within a bound. Moreover, the existing design method for stabilizing periodic switching signals requires the diagonal entries of system matrices of all subsystems to be strictly positive. In this study, we propose a novel approach to solve this problem completely. We construct a discrete-time Markov chain for a given DT-LCSS, termed the associated Markov chain, and prove the equivalence of stability and stabilizability between the DT-LCSS and the associated Markov chain. Based on this, verifiable necessary and sufficient conditions for stability and stabilizability are derived. Especially, the period of a stabilizing switching signal for an n-dimensional DT-LCSS can always be chosen within the bound n2−n+1. We propose a state-independent stabilizing switching signal design method for general stabilizable DT-LCSSs. We also prove the equivalence between stabilizability by state-independent switching laws and stabilizability by state-dependent switching laws. A state-dependent global stabilizing switching signal design method is also proposed. Additionally, the proposed results are applied to the consensus analysis of discrete-time leader–follower multi-agent systems with switching communication digraphs. The effectiveness of the theoretical results is demonstrated by examples.

Markovian Maintenance Planning of Ship Propulsion System Accounting for CII and System Degradation

The study’s objective is to create a method to select the best course of maintenance action for each state of ship propulsion system degradation while considering both the present and future costs and associated carbon intensity indicator, CII, rates. The method considers the effects of wind and wave action when considering fouling and ageing. The ship resistance in calm, wave, and wind conditions has been defined using standard operating models, which have also been used to estimate the required engine power, service speed, fuel consumption, generated CO2, CII, and subsequent maintenance costs. The maintenance takes into consideration the effects of profit loss because of lost opportunities and efficiency over time. Any maintenance choice has total costs associated with it, including extra fuel, upkeep, and missed opportunities. Using a discrete-time Markov chain, the ship’s propulsion system maintenance schedule is optimized. A decision has been reached regarding the specific maintenance measures to be undertaken for each state of the Markov chain among various alternatives. The choice of optimal maintenance is related to a Markov decision process and is made by considering both the current and future costs. The developed method can forecast the propulsion system’s future states and any required maintenance activities.

On the Interplay between Deadline-Constrained Traffic and the Number of Allowed Retransmissions in Random Access Networks.

In this paper, a network comprising wireless devices equipped with buffers transmitting deadline-constrained data packets over a slotted-ALOHA random-access channel is studied. Although communication protocols facilitating retransmissions increase reliability, a packet awaiting transmission from the queue experiences delays. Thus, packets with time constraints might be dropped before being successfully transmitted, while at the same time causing the queue size of the buffer to increase. To understand the trade-off between reliability and delays that might lead to packet drops due to deadline-constrained bursty traffic with retransmissions, the scenario of a wireless network utilizing a slotted-ALOHA random-access channel is investigated. The main focus is to reveal the trade-off between the number of retransmissions and the packet deadline as a function of the arrival rate. Towards this end, analysis of the system is performed by means of discrete-time Markov chains. Two scenarios are studied: (i) the collision channel model (in which a receiver can decode only when a single packet is transmitted), and (ii) the case for which receivers have multi-packet reception capabilities. A performance evaluation for a user with different transmit probabilities and number of retransmissions is conducted. We are able to determine numerically the optimal probability of transmissions and the number of retransmissions, given the packet arrival rate and the packet deadline. Furthermore, we highlight the impact of transmit probability and the number of retransmissions on the average drop rate and throughput.

Exponential synchronization of stochastic coupled neural networks with stochastic delayed impulsive effect

This paper studies the synchronization of stochastic coupled neural networks (SCNNs) with stochastic delayed impulses, which are more realistic and complex than the deterministic ones. Unlike previous works, we incorporate two distinct types of stochastic factors into the model: random noise affecting the systems continuous dynamics, and stochastic impulsive intensity applied at discrete impulsive instances. We extend the differential inequality with stochastic delayed impulses to stochastic systems without requiring that impulsive delay to be smaller than impulsive interval. We also model the impulsive intensity as a discrete-time Markov chain. By using the generalized inequality, graph theory, and stochastic analysis techniques, we derive a synchronization criterion for SCNNs with stochastic delayed impulses. We present numerical examples to demonstrate the effectiveness of our results.

Price Trend Forecasting in the Brazilian Stock Market using Discrete-Time Markov Chain

Price Trend Forecasting in the Brazilian Stock Market using Discrete-Time Markov Chain

Reliability assessment of discrete-time [formula omitted](G) retrial system based on different failure types and the [formula omitted]-shock model

The k/n(G) system is a typical reliability system in engineering practice. However, research on this system under discrete-time and retrial conditions is still limited. This paper establishes a k/n(G) retrial system model under a discrete-time condition based on different failure types and the δ-shock model, whose objective is to investigate its various reliability measures and the relationship between measures and parameters. Two methodologies are utilized in this paper to evaluate the system’s various measures: the analytical method for the discrete-time Markov process and the Monte Carlo simulation method. The two methodologies are compared, and optimization is performed on certain measures concerning system parameters. A discrete PH distribution representation method for component lifetime under the δ-shock is presented, the relationship between various reliability measures of the system and various parameters is determined, and a novel approach for system reliability research based on the discrete PH distribution and the δ-shock is provided in this paper.

Airfield pavement performance prediction using Clustered Markov Chain Models

This study pairs unsupervised learning methods of clustering and classification with discrete-time Markov Chain models for airport pavement performance prediction, with the goal of allowing the clustering algorithm to take the place of typical pre-analysis grouping of pavement sections. The influences of cluster method, number of clusters, input years, interval time span, and training data on prediction results are analysed. Results indicate that clustering using the deterioration profiles of Pavement Condition Index (PCI) before applying the Markov Chain is sufficient to reduce the error by half in predicted pavement conditions for most cases. The initial pavement degradation is a reliable source for clustering and future condition prediction using the appropriately matched Markov Chain. This approach minimises the data required to predict pavement condition and proves the early deterioration rate of a pavement section can be used to predict its overall lifespan.

Comparing the visual affordances of discrete time Markov chains and epistemic network analysis for analysing discourse connections

IntroductionResearchers in the learning sciences have been considering methods of analysing and representing group-level temporal data, particularly discourse analysis, in Computed Supported Collaborative Learning for many years.MethodsThis paper compares two methods used to analyse and represent connections in discourse, Discrete Time Markov Chains and Epistemic Network Analysis. We illustrate both methods by comparing group-level discourse using the same coded dataset of 15 high school students who engaged in group work. The groups were based on the tools they used namely the computer, iPad, or Interactive Whiteboard group. The aim here is not to advocate for a particular method but to investigate each method’s affordances.ResultsThe results indicate that both methods are relevant in evaluating the code connection within each group. In both cases, the techniques have supported the analysis of cognitive connections by representing frequent co-occurrences of concepts in a given segment of discourse.DiscussionAs the affordances of both methods vary, practitioners may consider both to gain insight into what each technique can allow them to conclude about the group dynamics and collaborative learning processes to close the loop for learners.

Arbitrary Modulation of Average Dwell Time in Discrete-Time Markov Chains Based on Tunneling Magnetoresistance Effect

Arbitrary Modulation of Average Dwell Time in Discrete-Time Markov Chains Based on Tunneling Magnetoresistance Effect

Stochastic viability in an island model with partial dispersal: Approximation by a diffusion process in the limit of a large number of islands

In this paper, we investigate a finite population undergoing evolution through an island model with partial dispersal and without mutation, where generations are discrete and non-overlapping. The population is structured into D demes, each containing N individuals of two possible types, A and B, whose viability coefficients, sA and sB, respectively, vary randomly from one generation to the next. We assume that the means, variances and covariance of the viability coefficients are inversely proportional to the number of demes D, while higher-order moments are negligible in comparison to 1/D. We use a discrete-time Markov chain with two timescales to model the evolutionary process, and we demonstrate that as the number of demes D approaches infinity, the accelerated Markov chain converges to a diffusion process for any deme size N≥2. This diffusion process allows us to evaluate the fixation probability of type A following its introduction as a single mutant in a population that was fixed for type B. We explore the impact of increasing the variability in the viability coefficients on this fixation probability. At least when N is large enough, it is shown that increasing this variability for type B or decreasing it for type A leads to an increase in the fixation probability of a single A. The effect of the population-scaled variances, σA2 and σB2, can even cancel the effects of the population-scaled means, μA and μB. We also show that the fixation probability of a single A increases as the deme-scaled migration rate increases. Moreover, this probability is higher for type A than for type B if the population-scaled geometric mean viability coefficient is higher for type A than for type B, which means that μA−σA2/2>μB−σB2/2.

The Prediction of Road Condition Value during Maintenance Based on Markov Process

The first step in the road handling effort is to survey to get an accurate road condition value to take correct action in implementing maintenance. As pavement performance is known to be probabilistic, various levels of uncertainty must be assumed. Modern pavement management methods are ineffective without an effective model to predict pavement performance. Discrete-time Markov chains are the most widely used probabilistic models, and examples from different countries worldwide can be found in pavement management systems. This research aims to predict the value of road conditions during maintenance and compare road assessments with real conditions during road maintenance using the IRI, SDI, and PCI methods using the Markov process. The analysis method used is to collect secondary data from related departments and carry out direct data collection in the field to obtain condition values based on IRI, SDI, and PCI to forecast by making a pavement condition prediction model based on the Markov process and then assessing road conditions by comparing the three index values with the slightest deviation value. The analysis showed that the average value of road conditions with the IRI indicator is 4.45, which is moderate, and the most negligible difference between the probability distribution of pavement condition prediction modeling and the actual survey results was the IRI (International Roughness Index) method. This model is closest to the actual conditions during implementation, with a difference value of 5.7%.

Deep neural network expressivity for optimal stopping problems

This article studies deep neural network expression rates for optimal stopping problems of discrete-time Markov processes on high-dimensional state spaces. A general framework is established in which the value function and continuation value of an optimal stopping problem can be approximated with error at most ε\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\varepsilon $\\end{document} by a deep ReLU neural network of size at most κdqε−r\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\kappa d^{\\mathfrak{q}} \\varepsilon ^{-\\mathfrak{r}}$\\end{document}. The constants κ,q,r≥0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\kappa ,\\mathfrak{q},\\mathfrak{r} \\geq 0$\\end{document} do not depend on the dimension d\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$d$\\end{document} of the state space or the approximation accuracy ε\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\varepsilon $\\end{document}. This proves that deep neural networks do not suffer from the curse of dimensionality when employed to approximate solutions of optimal stopping problems. The framework covers for example exponential Lévy models, discrete diffusion processes and their running minima and maxima. These results mathematically justify the use of deep neural networks for numerically solving optimal stopping problems and pricing American options in high dimensions.

Markov Chains to Explore the Nanosystems for the Biophysical Studies of Cancers

The immune response is essential for the human body to function well and to survive against the sudden and chronic diseases such as viral & bacterial infections and cancers. In the immunosurveillance process, Natural Killer (NK) cells are one of the main elements in controlling the development of such infections and, for this reason, they have become the subject of “in-depth” studies especially for the application of new forms of immunotherapy. NK cells can rapidly destroy both autologous and tumor cells in vitro and for this reason the interest in their function is increasingly growing. Their presence in the tumor micro-environment (TME) also assumes prognostic value since the repertoire of NK cell receptors has been linked to anti-tumor function. In this work, a Markov chain modeling approach is proposed to analyze the network of interactions that NK cells carry out with other immune elements in the defense against cancer such as CD4+ cells and CD8+ cells and dendritic cells (DCs) that activate and enhance immune responses. The probabilistic approach used is promising since it helps to understand the balance and the communication in the micro-environment, in a realistic manner. The advantage of discrete time Markov chain approach is that, it can be further extended to complex networks using the state-of-the-art algorithms and can also be translated for the novel AI tools for the cytokines and protein databases.

A measure of the importance of moment for ball-strike counts in a baseball plate appearance

ABSTRACT This study constructs a discrete-time Markov Chain (DTMC) model for a baseball plate appearance (PA) employing Major League Baseball’s pitch-by-pitch dataset. Based on the DTMC model, we propose a novel measure for a baseball PA, termed the Importance of Moment (IOM). The IOM quantifies the criticality of each ball-strike count situation, by assessing the probabilistic difference between the pitcher’s and hitter’s favourable outcomes (out vs reaching base). If the favours significantly vary right after a particular ball-strike count, then the count is deemed critical and is assigned a high IOM value. We empirically verify that IOM explains pitchers’ behaviour of fastball speed. We then further investigate whether the behaviour of ace pitchers differs significantly from the majority. Several interesting properties are found from the analysis. Firstly, the path independence assumption generally holds, with the exception of the ball-strike count of 2B1S. Second, pitchers tend to throw the faster fastball at counts with higher IOM values. Lastly, ace pitchers are capable of pitching even faster fastball in two-strike situations in which IOM is high. The DTMC effectively models the probabilistic structure of a baseball PA, and the proposed IOM measure serves as a useful tool for explaining player behaviour.