Discrete-time Markov Chain Model Research Articles

Airfield pavement performance prediction using Clustered Markov Chain Models

This study pairs unsupervised learning methods of clustering and classification with discrete-time Markov Chain models for airport pavement performance prediction, with the goal of allowing the clustering algorithm to take the place of typical pre-analysis grouping of pavement sections. The influences of cluster method, number of clusters, input years, interval time span, and training data on prediction results are analysed. Results indicate that clustering using the deterioration profiles of Pavement Condition Index (PCI) before applying the Markov Chain is sufficient to reduce the error by half in predicted pavement conditions for most cases. The initial pavement degradation is a reliable source for clustering and future condition prediction using the appropriately matched Markov Chain. This approach minimises the data required to predict pavement condition and proves the early deterioration rate of a pavement section can be used to predict its overall lifespan.

A measure of the importance of moment for ball-strike counts in a baseball plate appearance

ABSTRACT This study constructs a discrete-time Markov Chain (DTMC) model for a baseball plate appearance (PA) employing Major League Baseball’s pitch-by-pitch dataset. Based on the DTMC model, we propose a novel measure for a baseball PA, termed the Importance of Moment (IOM). The IOM quantifies the criticality of each ball-strike count situation, by assessing the probabilistic difference between the pitcher’s and hitter’s favourable outcomes (out vs reaching base). If the favours significantly vary right after a particular ball-strike count, then the count is deemed critical and is assigned a high IOM value. We empirically verify that IOM explains pitchers’ behaviour of fastball speed. We then further investigate whether the behaviour of ace pitchers differs significantly from the majority. Several interesting properties are found from the analysis. Firstly, the path independence assumption generally holds, with the exception of the ball-strike count of 2B1S. Second, pitchers tend to throw the faster fastball at counts with higher IOM values. Lastly, ace pitchers are capable of pitching even faster fastball in two-strike situations in which IOM is high. The DTMC effectively models the probabilistic structure of a baseball PA, and the proposed IOM measure serves as a useful tool for explaining player behaviour.

Analyzing the impact of cyber-attacks on the performance of digital twin-based industrial organizations

Digital twin is a virtual copy of their “real-world counterparts”, which refer to physical objects, systems, or entities that exist in the physical world. They share valuable information about how the system works by connecting the replica with the real product via the internet and sensors. In digital twin organizations, industrial control systems and the internet are the main sources of data collection and synchronization, resulting in an increase in cyber-attacks. As a result, they could cause data leaks and be used by hackers to launch attacks without being detected. Researching the appropriate defenses against such attacks is of paramount interest. To that end, in this study, we build an integrated model utilizing the combination of a Dynamic Bayesian Network (DBN) and a Discrete-Time Markov Chain (DTMC) to analyze the considerable consequences of cyberattacks on a digital twin-based organization over different time period. DTMC models recovery and vulnerability for DT-based organizations, and then it is paired with the DBN model to simulate the cyberattack propagation behavior in the organization's IT performance. Next, we calculate the organization's work level and total expected utility under two different cyber-attack scenarios to determine its IT performance and financial behavior. Finally, we evaluate the organization's IT system's reliability and resilience after implementing two prevention mechanisms. Result indicates that Distributed Denial-of-Service (DDoS) cyberattack can cause more destruction to the digital twin organization's IT system than a malware attack in terms of cyberattack propagation, work and utility level, reliability, and resilience capability. The findings of the analysis hold practical applicability in real-world settings, enabling the identification of high-risk cyberattacks within DT-based organizations, the analysis and prediction of an organization's IT performance during actual cyberattacks, and the formulation of effective prevention strategies to address cyber security concerns within DT-based organizations.

Qualitative analysis of solutions for a degenerate partial differential equations model of epidemic spread dynamics

Compartmental models are widely used in mathematical epidemiology to describe the dynamics of infectious diseases or in mathematical models of population genetics. In this study, we study a time-dependent Susceptible-Infectious-Susceptible (SIS) Partial Differential Equation (PDE) model that is based on a diffusion-drift approximation of a probability density from a well-known discrete-time Markov chain model. This SIS-PDE model is conservative due to the degeneracy of the diffusion term at the origin. The main results of this article are the qualitative behavior of weak solutions, the dependence of the local asymptotic property of these solutions on initial data, and the existence of Dirac delta function type solutions. Moreover, we study the long-term behavior of solutions and confirm our analysis with numerical computations.

High-frequency stock market order transitions during the US-China trade war 2018: A discrete-time Markov chain analysis.

Statistical analysis of high-frequency stock market order transaction data is conducted to understand order transition dynamics. We employ a first-order time-homogeneous discrete-time Markov chain model to the sequence of orders of stocks belonging to six different sectors during the US-China trade war of 2018. The Markov property of the order sequence is validated by the Chi-square test. We estimate the transition probability matrix of the sequence using maximum likelihood estimation. From the heatmap of these matrices, we found the presence of active participation by different types of traders during high volatility days. On such days, these traders place limit orders primarily with the intention of deleting the majority of them to influence the market. These findings are supported by high stationary distribution and low mean recurrence values of add and delete orders. Further, we found similar spectral gap and entropy rate values, which indicates that similar trading strategies are employed on both high and low volatility days during the trade war. Among all the sectors considered in this study, we observe that there is a recurring pattern of full execution orders in the Finance & Banking sector. This shows that the banking stocks are resilient during the trade war. Hence, this study may be useful in understanding stock market order dynamics and devise trading strategies accordingly on high and low volatility days during extreme macroeconomic events.

Developing a Two-Stage Supply Chain Model using a Discrete Time Markov Chain Model during Supply Chain Disruptions

Developing a Two-Stage Supply Chain Model using a Discrete Time Markov Chain Model during Supply Chain Disruptions

ECG Modeling for Simulation of Arrhythmias in Time-Varying Conditions.

The present article proposes an ECG simulator that advances modeling of arrhythmias and noise by introducing time-varying signal characteristics. The simulator is built around a discrete-time Markov chain model for simulating atrial and ventricular arrhythmias of particular relevance when analyzing atrial fibrillation (AF). Each state is associated with statistical information on episode duration and heartbeat characteristics. Statistical, time-varying modeling of muscle noise, motion artifacts, and the influence of respiration is introduced to increase the complexity of simulated ECGs, making the simulator well suited for data augmentation in machine learning. Modeling of how the PQ and QT intervals depend on heart rate is also introduced. The realism of simulated ECGs is assessed by three experienced doctors, showing that simulated ECGs are difficult to distinguish from real ECGs. Simulator usefulness is illustrated in terms of AF detection performance when either simulated or real ECGs are used to train a neural network for signal quality control. The results show that both types of training lead to similar performance.

Modeling of emergency evacuation in high rise buildings considering congestion at stairs based on Markov chains

High-rise buildings are vulnerable to encountering fires or other emergencies at any time, requiring an immediate evacuation of occupants. In the context of building evacuations, various engineering parameters are often utilized to evaluate the time required for people to reach a safe place when descending a staircase. A discrete-time Markov chain model is proposed to evaluate the evacuation time of the occupants in such circumstances by taking into consideration factors like merging at the stairs and congestion on the floor. Based on the congestion probabilities with respect to the number of evacuees in the building and other human behavior factors that slow down the rate of evacuation, non-congested and congested scenarios are evaluated. The effect of the critical parameters that influenced the evacuation process in the high-rise building, such as walking speed, number of evacuees, and effective stair width, is evaluated in both scenarios. The findings indicate an optimal evacuation speed of 0.7–1.4 m/s. The effective width of the stair is suggested to be between 1.6 and 2.4 m using the model. The simulation results are in good accordance with the SFPE method, the NIST egress simulator, and the agent-based model results.

Opportunistic Routing-Aided Cooperative Communication Network With Energy Harvesting

In this article, a cooperative communication network based on energy-harvesting (EH) decode-and-forward (DF) relays that harvest energy from the ambience using buffers with the harvest-store-use (HSU) architecture is considered. An opportunistic routing (OR) protocol, which selects the transmission path of packet based on the node transmission priority, is proposed to improve data delivery in this network. Additionally, an algorithm based on the state transition matrix (STM) is proposed to obtain the probability distribution of the candidate broadcast node set. Based on the probability distribution, the existence conditions and the theoretical expressions for the limiting distribution of energy in energy buffers using a discrete-time continuous-state space Markov chain (DCSMC) model are derived. Furthermore, the closed-form expressions for network outage probability and throughput are obtained with the help of the limiting distributions of energy stored in buffers. Numerous experiments have been performed to validate the derived theoretical expressions of the performance of this cooperative communication network.

Performance Modeling and Analysis of BASUR-CARQ Protocol Based on Interrelay Interference Cancellation

To recover the performance loss of nodes in half-duplex mode, this paper proposes a buffer-aided successive relay protocol based on cooperative automatic repeat request mechanism (BASUR-CARQ). Based on the fact that interrelay interference (IRI) will occur when relay nodes transmit or receive simultaneously, an interference cancellation operator is proposed to determine whether the interference is eliminated to reduce the outage probability. Moreover, a delay model for data frame transmission is proposed based on CARQ mechanism, and a closed-form expression for the average delay is derived. A 6-state discrete-time Markov chain (DTMC) model is developed to obtain the system throughput, and a closed-form expression for the system energy efficiency under M-ary modulation is derived. Finally, the simulation results show that with the setting of parameters that can balance the main performance, the delay performance of BASUR-CARQ protocol is significantly enhanced compared to the traditional protocols, and the throughput of BASUR-CARQ protocol is also optimized at high signal-to-noise ratio (SNR). Meanwhile, the energy efficiency of BASUR-CARQ protocol is significantly improved for the successive relay communication system without interference cancellation technique.

Mobile agent path planning under uncertain environment using reinforcement learning and probabilistic model checking

The major challenge in mobile agent path planning, within an uncertain environment, is effectively determining an optimal control model to discover the target location as quickly as possible and evaluating the control system’s reliability. To address this challenge, we introduce a learning-verification integrated mobile agent path planning method to achieve both the effectiveness and the reliability. More specifically, we first propose a modified Q-learning algorithm (a popular reinforcement learning algorithm), called QEA−learning algorithm, to find the best Q-table in the environment. We then determine the location transition probability matrix, and establish a probability model using the assumption that the agent selects a location with a higher Q-value. Secondly, the learnt behaviour of the mobile agent based on QEA−learning algorithm, is formalized as a Discrete-time Markov Chain (DTMC) model. Thirdly, the required reliability requirements of the mobile agent control system are specified using Probabilistic Computation Tree Logic (PCTL). In addition, the DTMC model and the specified properties are taken as the input of the Probabilistic Model Checker PRISM for automatic verification. This is preformed to evaluate and verify the control system’s reliability. Finally, a case study of a mobile agent walking in a grids map is used to illustrate the proposed learning algorithm. Here we have a special focus on the modelling approach demonstrating how PRISM can be used to analyse and evaluate the reliability of the mobile agent control system learnt via the proposed algorithm. The results show that the path identified using the proposed integrated method yields the largest expected reward.

Opportunistic Routing Aided Cooperative Communication MRC Network With Energy-Harvesting Nodes

In this research, we present a cooperative communication network based on two energy-harvesting (EH) decode-and-forward (DF) relays that contain the harvest-store-use (HSU) architecture and may harvest energy from the surrounding environment by utilizing energy buffers. To improve the performance of this network, an opportunistic routing (OR) algorithm is presented that takes into account channel status information, relay position, and energy buffer status, as well as using the maximal ratio combining (MRC) at the destination to combine the signals received from the source and relays. The theoretical expressions for limiting distribution of energy stored in infinite-size buffers are derived from the discrete-time continuous-state space Markov chain model (DCSMC). In addition, the theoretical expressions for outage probability, throughput, and per-packet time slot cost in the network are obtained utilizing both the limiting distributions of energy buffers and the probabilities of transmitter candidates set. Through numerous simulation results, it is demonstrated that simulation results match with corresponding theoretical results.

A Markov chain-based IoT system for monitoring and analysis of urban air quality.

Severe deterioration of urban air quality in Asian cities is the cause of a large number of deaths every year. A Markov chain-based IoT system is developed in this study to monitor, analyze, and predict urban air quality. The proposed sensing setup is integrated with an automobile and is used for collecting air quality information. An Android application is used to transfer and store the sensed data in the data cloud. The data stored is used to generate the transition matrix of the AQI states and calculate return periods for each AQI state. The estimated time interval after which an AQI event recurs or is repeated is known as return period. The actual return periods for each AQI state at the test locations in Delhi-NCR are compared with those predicted using discrete time Markov chain (DTMC) models. Average absolute forecast error using our model was found to be 3.38% and 4.06%, respectively, at the selected locations.

A Markov Chain Approach to the Pattern of Blood Donation Status at a Blood Service Centre in Zimbabwe

Background: Blood donors’ behaviour towards blood donation is not easily predictable and can be considered a stochastic random variable. A four-state Markov chain technique was defined and adopted in this study. The transition probabilities of blood donation within the four identified states, viz: new, regular, occasional, and lapsed donors were used to making further inferences about the dynamics in blood donation in Harare, Zimbabwe. Objectives: The paper presents a four-state Discrete Time Markov Chain (DTMC) model in analysing the changes in blood donation status over the four-year study period. Methodology: A transition probabilities matrix was developed and parameters estimated using the maximum likelihood method and two other approaches, and inferences were made based on the resultant transition matrix. Results: About 56% of new donors made at least one repeat donation and became regular donors within the first year, and the numbers gradually declined with time, whilst the lapsed donors increased from 35.6% in the second year to 55.6% in year 4. The long-run probabilities tell the same, with 80.9% of blood donations becoming lapsed in the long run. Depending on the current state of donation, new or regular donations will likely move to the regular donation state in the following time step (year). On the other end, occasional and lapsed donations have a higher probability of entering the lapsed donation state in the following time step (year). Conclusion: The paper provides useful insights into the Markovian transition probabilities among the blood donation states, and this has implications on future blood donors’ pool and blood bank inventory in Zimbabwe. The decline in the number of donors who make repeat donations is a worrisome trend since regular donations are the lifeline of any blood service centre.

The multilevel hierarchical data EM-algorithm. Applications to discrete-time Markov chain epidemic models

The theory of multilevel hierarchical data Expectation Maximization (EM)-algorithm is introduced via discrete time Markov chain (DTMC) epidemic models. A general model for a multilevel hierarchical discrete data is derived. The observed sample Y in the system is a stochastic incomplete data, and the missing data Z exhibits a multilevel hierarchical data structure. The EM-algorithm to find ML-estimates for parameters in the stochastic system is derived. Applications of the EM-algorithm are exhibited in the two DTMC models, to find ML-estimates of the system parameters. Numerical results are given for influenza epidemics in the state of Georgia (GA), USA.

Context-aware IoT-enabled framework to analyse and predict indoor air quality

• An IoT-based framework for data collection and context modelling has been developed. • Extended Kalman Filter (EKF) approach is used to deal with inaccuracies, missing data and eliminate errors. • The EKF-derived indoor pollutant concentrations are employed in context reasoning for proposing a new index. • An adaptive neuro-fuzzy inference system (ANFIS) uses the percent of dissatisfied people (PPD), ventilation rate (VR), and AQI data to identify the present state of IAQ and the percentage of time when the air quality in the classroom is unhealthy. For a productive and healthy life, air quality plays an important role. This paper addresses the requirements to develop a system capable of providing real-time information, predictions, and alerts about the indoor environment using context-awareness. The proposed IoT system serves for data collection, pre-processing, defining rules, and forecasting the predicting states of the indoor environment by giving information to the end-user about the alerts and recommendations. A novel approach based on the indoor pollutants T, RH, CO 2 , PM 2.5 , PM 10 , and CO for the determination of the status of the environment is proposed. The pre-processing is used for filtering data using and extended Kalman filter. Further, the system uses an adaptive neuro-fuzzy inference system (ANFIS) and discrete-time Markov chains (DTMC) to predict the state of the indoor environment with the help of daily air pollution concentrations and environmental parameters. The ANFIS model predictor considers the value of indoor pollutants to form a new index: State of indoor air (SIA). For analysis and forecasting of the new index SIA, the DTMC model is used. The collected and measured data is stored in the IoT cloud using the sensing setup, and sensed information is used to develop the SIA transfer matrix, generating return durations corresponding to each SIA and providing alerts based on the data to the end-user. The models are assessed using the expected and actual return durations. The most frequent interior ventilation states, according to the predictions, are poor and moderate. Only 0.08 percent of the time does the IAQ remain in a good state. Two-thirds of the time (66%), the indoor ventilation is severe (poor, very poor, or hazardous); 19% of the time it is very bad, and 15% of the time it is hazardous, suggesting and warning that there is a very high probability of unhealthy AQI in educational institutions in the Delhi-NCR region. Performance is measured by the comparison between actual and forecasted return periods, and the forecast error for our system is low, with an absolute forecast error of 3.47% on an average.

Modeling and Performance Analysis of ECS-HTCARQ Protocol in Interference Scenario

To extend the lifetime of energy-constrained wireless networks, this paper proposes a three-node model of the wireless-powered cooperative communication network (WPCCN). The model consists of an energy-constrained source node, an energy-sufficient relay node, and a destination node and considers two different cochannel interference (CCI) sources at the relay and destination nodes. To ensure the reliability of data transmission, we introduce the cooperative automatic retransmission request (CARQ) protocol, propose the energy-constrained source that harvests and cooperates automatic retransmission request (ECS-HTCARQ) protocol in interference scenario, derive a closed-form expression for the outage probability under Rayleigh fading channels, and establish a discrete-time Markov chain (DTMC) model to analyze the system throughput. The relationship between parameters such as the energy harvesting time allocation factor and energy harvesting efficiency on the outage probability and the system throughput is obtained. Numerical results show that the difference in system throughput is small as the increasing number of transmissions at a high signal-to-interference ratio (SIR), but there is a significant reduction in the outage probability. Finally, the optimal value of the energy harvesting time allocation factor is given under this model.

Stochastic approach to Markovian interrelationship assessment of solar activity indices

This paper employs a discrete-time Markov chain (DTMC) stochastic process to investigate a state/event based Markovian interrelationship between various solar activity indices (SAI) (including 10.7 cm solar radio flux (SF10.7), coronal index (CI), solar flare index (SFI) and total solar irradiance (TSI)) in relation to sunspot number (SSN). First, we applied the first order DTMC model as a first approximation to the total number of transitions between different states of SAI in order to estimate the probability of occurrence corresponding with each transition. Next, several DTMC descriptors like persistency, state dependency, stationarity, mean first passage time and entropy are derived from estimated transition probability matrices. These descriptors are very useful as they related to time series characteristics (like randomness, nature of cycles and predictability) within a stochastic dynamical system as well as crucial for checking the applicability of Markov chain method. Therefore, via the DTMC analysis and derived descriptors, this study found remarkable similarities in the formation of transition matrices and diagrams, significant 2-dimensional correlation values, robust self-communication behaviour among states, existence of dependent successive transitions and stationary nature of data throughout the space. Further, the resemblance in the average transit time from one state to another, probabilistically disordered symmetrical time series and existence of randomness in transition states has been observed. Therefore, results obtained in this paper provide a new insight to increase the level of knowledge of the possible linkage between underlying SAI that could be helpful in enhanced understanding of the potential future climate changes and other solar energy-related objectives.

An Analytical Model for the Aggregate Throughput of IEEE 802.11ah Networks under the Restricted Access Window Mechanism.

The IEEE 802.11ah is an amendment to the IEEE 802.11 standard to support the growth of the Internet of Things (IoT). One of its main novelties is the restricted access window (RAW), which is a channel access feature designed to reduce channel contention by dividing stations into RAW groups. Each RAW group is further divided into RAW slots, and stations only attempt channel access during the RAW slot they were assigned to. In this paper, we propose a discrete-time Markov chain model to evaluate the average aggregate throughput of IEEE 802.11ah networks using the RAW mechanism under saturated traffic and ideal channel conditions. The proposed analytical model describes the behavior of an active station within its assigned RAW slot. A key aspect of the model is the consideration of the event of RAW slot time completion during a station’s backoff operation. We study the average aggregate network throughput for various numbers of RAW slots and stations in the network. The numerical results derived from our analytical model are compared to computer simulations based on an IEEE 802.11ah model developed for the ns-3 simulator by other researchers, and its performance is also compared to two other analytical models proposed in the literature. The presented results indicate that the proposed analytical model reaches the closest agreement with independently-derived computer simulations.

Collision-Based Window-Scaled Back-Off Mechanism for Dense Channel Resource Allocation in Future Wi-Fi

Wireless local area networks (WLANs), known as Wi-Fi, are widely deployed to meet the enhanced needs of data-centric internet applications, such as wireless docking, unified communications, cloud computing, interactive multimedia gaming, progressive streaming, support of wearable devices, up-link broadcasts and cellular offloading. Wi-Fi networks typically adopt the Distributed Coordination Function (DCF)-based Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA), which uses the Binary Exponential Back-off (BEB) algorithm at the MAC layer mechanism to access channel resources. Currently deployed Wi-Fi networks face huge challenges towards efficient channel access for denser environments due to the blind exponential increase/decrease of a contention window (CW) procedure that is inefficient for a higher number of contending stations. Several modifications and amendments have been proposed to improve the performance of the MAC layer channel access based on a fixed or variable CW size. However, a more realistic network density-based channel resource allocation solution is still missing. An efficient channel resource allocation is one of the most critical challenges for future highly dense WLANs, such as High-Efficiency WLAN (HEW). In this paper, we propose a Channel Collision-based Window Scaled Back-off (CWSB) mechanism for channel resource allocation in HEW. In our proposed CWSB, all contending stations select an optimized CW size for each back-off stage for collided or successfully transmitted data frames. We affirm the performance of the proposed CWSB mechanism with the help of an Iterative Discrete Time Markov Chain (I-DTMC) model. This paper evaluates the performance of our proposed CWSB mechanism in HEW Wi-Fi networks using an NS3 simulator in terms of the normalized throughput and channel access delay compared to the state-of-the-art BEB and a recently proposed mechanism.