Multi-dimensional Markov Chain Research Articles

Down/up crossing properties of weighted Markov collision processes

This paper concentrates on considering the down/up crossing property of weighted Markov collision processes. The joint probability generating function of down crossing and up crossing numbers of weighted Markov collision processes until its extinction are obtained by constructing and studying a related multi-dimensional Markov chain. Hence, the joint probability distribution of down crossing and up crossing numbers and the mean numbers are obtained.

Mathematical Model of Operation of a Cell of a Mobile Communication Network With Adaptive Modulation Schemes and Handover of Mobile Users

A model of a cell of a communication network, divided into zones, with the dependence of the users' service time on the zone, in which they are located, is considered. The arrival flow of users is defined by a marked Markovian process. The number of users that can receive service in a cell simultaneously is finite. If the number of users reached the limit, then new users are lost, except the users that come to the cell already during the service (handover users). For the short-term storage of such users, there is a buffer of a finite capacity. Cell operation is modeled by a multi-server queueing system. The behavior of this system is described by a multi-dimensional Markov chain. The algorithms for calculating the stationary distribution of this chain and the main characteristics of the quality of service have been developed.

Priority Multi-Server Queueing System with Heterogeneous Customers

In this paper, we analyze a multi-server queueing system with heterogeneous customers that arrive according to a marked Markovian arrival process. Customers of two types differ in priorities and parameters of phase type distribution of their service time. The queue under consideration can be used to model the processes of information transmission in telecommunication networks in which often the flow of information is the superposition of several types of flows with correlation of inter-arrival times within each flow and cross-correlation. We define the process of information transmission as the multi-dimensional Markov chain, derive the generator of this chain and compute its stationary distribution. Expressions for computation of various performance measures of the system, including the probabilities of loss of customers of different types, are presented. Output flow from the system is characterized. The presented numerical results confirm the high importance of account of correlation in the arrival process. The values of important performance measures for the systems with the correlated arrival process are essentially different from the corresponding values for the systems with the stationary Poisson arrival process. Measurements in many real world systems show poor approximation of real flows by such an arrival process. However, this process is still popular among the telecommunication engineers due to the evident existing gap between the needs of adequately modeling the real life systems and the current state of the theory of algorithmic methods of queueing theory.

A Priority Queue with Many Customer Types, Correlated Arrivals and Changing Priorities

A single-server queueing system with a finite buffer, several types of impatient customers, and non-preemptive priorities is analyzed. The initial priority of a customer can increase during its waiting time in the queue. The behavior of the system is described by a multi-dimensional Markov chain. The generator of this chain, having essential dependencies between the components, is derived and formulas for computation of the most important performance indicators of the system are presented. The dependence of some of these indicators on the capacity of the buffer space is illustrated. The profound effect of the phenomenon of correlation of successive inter-arrival times and variance of the service time is numerically demonstrated. Results can be used for the optimization of dispatching various types of customers in information transmission systems, emergency departments and first aid stations, perishable foods supply chains, etc.

Analysis of retrial queue with heterogeneous servers and Markovian arrival process

Multi-server retrial queueing system with heterogeneous servers is analyzed. Requests arrive to the system according to the Markovian arrival process. Arriving primary requests and requests retrying from orbit occupy an available server with the highest service rate, if there is any available server. Otherwise, the requests move to the orbit having an infinite capacity. The total retrial rate infinitely increases when the number of requests in orbit increases. Service periods have exponential distribution. Behavior of the system is described by multi-dimensional continuous-time Markov chain which belongs to the class of asymptotically quasi-toeplitz Markov chains. This allows to derive simple and transparent ergodicity condition and compute the stationary probabilities distribution of chain states. Presented numerical results illustrate the dynamics of some system effectiveness indicators and the importance of considering of correlation in the requests arrival process.

Competitive queueing systems with comparative rating dependent arrivals

We consider the joint operation of two multi-server queueing systems. Both systems provide the same type of service and compete for customers. It is assumed that the first system is controlled by ourselves while the second one is controlled by our competitor. The arriving customers are shared between the systems depending on the comparative rating of our system. The stationary behavior of the considered model under the fixed parameters of both systems and the fixed mechanism of calculation of the rating is described by a continuous-time multi-dimensional Markov chain. The stationary distribution of this Markov chain is computed. Expressions for the key performance measures of the systems are derived. Obtained results provide an opportunity to analyse possible consequences of various managerial actions aiming to maximize the profit of our system. A numerical experiment illustrates the application of the results for making a decision about the rationality of establishing and maintaining a service system while an alternative system providing the same type of service already exists.

Tighter Lyapunov Truncation for Multi-Dimensional Continuous Time Markov Chains with Known Moments

Continuous Time Markov chains (CTMCs) are widely used to model and analyze networked systems. A common analysis approach is to solve the system of balance equations governing the state transitions of a CTMC to obtain its steadystate probability distribution, and use the state probabilities to derive or compute various performance measures. In many systems, the state space of the underlying CTMC is infinite and multi-dimensional with state-dependent transitions; exact analysis of such models is challenging. For example, the exact probability distribution of the number of jobs in the Discriminatory Processor Sharing (DPS) system, first proposed by Kleinrock in 1967 [4], is still an open challenge. Likewise, obtaining the exact state probabilities of quasi-birth-and-death (QBD) processes with leveldependent transitions is known to be challenging [1]; QBDs are infinite state space multi-dimensional Markov chains in which states are organized into levels and transitions are skip-free between the levels.

Effective algorithm for computation of the stationary distribution of multi-dimensional level-dependent Markov chains with upper block-Hessenberg structure of the generator

Multi-dimensional level-dependent Markov chains with the upper block-Hessenberg structure of the generator have found extensive applications in applied probability for solving the problems of queueing, reliability, inventory, etc. However, the problem of computing the stationary distribution of such chains is not completely solved. There is a known algorithm for multi-dimensional Asymptotically Quasi-Toeplitz Markov Chains, but, it is required a large amount of computer resources and time-consuming. In this paper, we propose a new effective algorithm that is much less time- and memory-consuming. The new algorithm can be used for analyzing any multi-dimensional Markov chain with the considered structure of the generator. To numerically demonstrate the advantages of this algorithm over the known one, we use it for analysis of a novel single-server retrial queueing system with the batch Markovian arrival process (BMAP), a finite buffer, non-persistent customers and an unreliable server. We derive a transparent ergodicity condition for this queueing system. Then, assuming that this condition is fulfilled, we apply the new algorithm and demonstrate its advantages over the known one.

Analysis of a Semi-Open Queuing Network with a State Dependent Marked Markovian Arrival Process, Customers Retrials and Impatience

We consider a queuing network with single-server nodes and heterogeneous customers. The number of customers, which can obtain service simultaneously, is restricted. Customers that cannot be admitted to the network upon arrival make repeated attempts to obtain service. The service time at the nodes is exponentially distributed. After service completion at a node, the serviced customer can transit to another node or leave the network forever. The main features of the model are the mutual dependence of processes of customer arrivals and retrials and the impatience and non-persistence of customers. Dynamics of the network are described by a multidimensional Markov chain with infinite state space, state inhomogeneous behavior and special structure of the infinitesimal generator. The explicit form of the generator is derived. An effective algorithm for computing the stationary distribution of this chain is recommended. The expressions for computation of the key performance measures of the network are given. Numerical results illustrating the importance of the account of the mentioned features of the model are presented. The model can be useful for capacity planning, performance evaluation and optimization of various wireless telecommunication networks, transportation and manufacturing systems.

Modelling and Performance Analysis of Wireless LAN Enabled by RF Energy Transfer

RF signals can be relied upon for transferring energy to those power-thirsty wireless devices. Thanks to the broadcast nature of wireless channels, dedicated RF signals for delivering information to specific devices can also be received by others for energy harvesting. Coordinating the conventional wireless information transfer and the wireless energy transfer (WET) requires a systematic design in all OSI layers, which yields data and energy integrated communication networks. Based on the classic carrier-sense-multiple-access with collision avoidance protocol, we originally propose a distributed multiple access protocol in the medium access control (MAC) layer for the indoor wireless-local-area-network (WLAN) powered by RF signal-based WET. The operation of our proposed protocol in the WET powered indoor WLAN is modeled by a multi-dimensional Markov chain, which is exploited for deriving the closed-form throughput of the WLAN studied. The simulation results demonstrate the accuracy of our theoretical analysis, which pave the way for the future optimization design in the MAC layer of WET powered indoor WLAN.

Semantic-Enhanced Multi-Dimensional Markov Chains on Semantic Trajectories for Predicting Future Locations.

In this work, we investigate the performance of Markov Chains with respect to modelling semantic trajectories and predicting future locations. In the first part, we examine whether and to what degree the semantic level of semantic trajectories affects the predictive performance of a spatial Markov model. It can be shown that the choice of the semantic level when describing trajectories has a significant impact on the accuracy of the models. High-level descriptions lead to better results than low-level ones. The second part introduces a multi-dimensional Markov Chain construct that considers, besides locations, additional context information, such as time, day and the users’ activity. While the respective approach is able to outperform our baseline, we could also identify some limitations. These are mainly attributed to its sensitivity towards small-sized training datasets. We attempt to overcome this issue, among others, by adding a semantic similarity analysis component to our model that takes the varying role of locations due each time to the respective purpose of visiting the particular location explicitly into consideration. To capture the aforementioned dynamics, we define an entity, which we refer to as Purpose-of-Visit-Dependent Frame (PoVDF). In the third part of this work, we describe in detail the PoVDF-based approach and we evaluate it against the multi-dimensional Markov Chain model as well as with a semantic trajectory mining and prefix tree based model. Our evaluation shows that the PoVDF-based approach outperforms its competition and lays a solid foundation for further investigation.

Multi-threshold control by a single-server queuing model with a service rate depending on the amount of harvested energy

We consider a single-server queuing model with energy harvesting. Customers and energy units arrive in the marked Markovian arrival flow. Buffers for customers and energy are finite. There are several service modes distinguished by the number of required units of energy, the service rate and the quality of service. The choice of the service mode for the next customer depends on the current number of energy units in the system. The quality of system operation is assumed to be defined by a customer’s loss probability. Loss can occur due to buffer overflow, low-quality service or impatience during the waiting time. The control strategy is of a multi-threshold type with thresholds defined by the amount of available energy. Under the fixed values of the thresholds, the behaviour of the system is described by a multi-dimensional Markov chain. The generator of this chain is derived. The main performance measures are calculated. The distribution of the waiting time is derived in explicit form. A numerical study of the model via extensive experiments that provide insight into the quantitative behaviour of the system is presented.

Procedural Content Generation via Machine Learning (PCGML)

This survey explores procedural content generation via machine learning (PCGML), defined as the generation of game content using machine learning models trained on existing content. As the importance of PCG for game development increases, researchers explore new avenues for generating high-quality content with or without human involvement; this paper addresses the relatively new paradigm of using machine learning (in contrast with search-based, solver-based, and constructive methods). We focus on what is most often considered functional game content, such as platformer levels, game maps, interactive fiction stories, and cards in collectible card games, as opposed to cosmetic content, such as sprites and sound effects. In addition to using PCG for autonomous generation, cocreativity, mixed-initiative design, and compression, PCGML is suited for repair, critique, and content analysis because of its focus on modeling existing content. We discuss various data sources and representations that affect the generated content. Multiple PCGML methods are covered, including neural networks: long short-term memory networks, autoencoders, and deep convolutional networks; Markov models: $n$-grams and multi-dimensional Markov chains; clustering; and matrix factorization. Finally, we discuss open problems in PCGML, including learning from small data sets, lack of training data, multilayered learning, style-transfer, parameter tuning, and PCG as a game mechanic.

Soft-Sensing CQI Feedback-Based Access Scheme in Cognitive Radio Networks

In this paper, an access scheme for cognitive radio networks is proposed in which secondary users (SUs) make use of the primary user(s) (PUs) channel quality feedback information. SUs decide their transmission strategy based on the PUs’ channel quality indicator (CQI) feedback available in the PU network and the SUs’ sensing of PUs’ activity. We consider two energy sensing approaches, namely, hard-sensing, and soft-sensing. An SU accesses the primary channel with access probabilities which are selected to maximize the SU service rate while ensuring PUs’ quality of service requirements defined as primary queues’ stability. The proposed system is modeled using a multi-dimensional Markov chain. Closed-form expressions for secondary users’ throughput and the average primary users’ packet delays are derived. Moreover, SUs’ access probabilities selection is formulated as a constrained optimization problem. Performance of the proposed scheme, in terms of the secondary user service rate and the primary user delay, is shown to be superior to other systems in which secondary users do not leverage PUs’ CQI feedback information. 1 1 Part of this paper was published in [1] .

An Analytical Framework for IEEE 802.15.6-Based Wireless Body Area Networks With Instantaneous Delay Constraints and Shadowing Interruptions

In this paper, a novel queueing analytical framework is proposed to evaluate the performance of IEEE 802.15.6-based carrier-sense multiple access with collision avoidance (CSMA/CA) scheduling in wireless body area networks with joint consideration of instantaneous delay constraints and body shadowing effects. Specifically, we develop an absorbing Markov chain to model the medium access process with error controls that is defined by the IEEE standard and design a random time-limited single vacation to describe the potential body shadowing interruption process. To guarantee the timeliness of received packets and avoid energy waste for transmitting valueless packets, an instantaneous delay constraint is carefully considered, which is then characterized by an overdeadline packet dropping process with a predefined waiting deadline. In our analysis, a Markovian arrival process is also adopted to capture the correlation of arrival traffic, and all other random processes are modeled by phase-type distributions, which make our framework more general and comprehensive. To address the inherent complexity of the original model, based on the transient queueing analysis, we develop a buffer-overflowing queueing principle to approximate the overdeadline packet dropping principle by solving a buffer length optimization problem. After that, we construct a multidimensional discrete-time Markov chain to analyze the stationary distribution through the matrix-geometric method. Performance measures, including the average delay, waiting time distribution, and packet transmission failure probability, are derived. The accuracy of our proposed analytical framework is validated by extensive simulations.

Oil-gas reservoir lithofacies stochastic modeling based on one- to three-dimensional Markov chains

Stochastic modeling techniques have been widely applied to oil-gas reservoir lithofacies. Markov chain simulation, however, is still under development, mainly because of the difficulties in reasonably defining conditional probabilities for multi-dimensional Markov chains and determining transition probabilities for horizontal strike and dip directions. The aim of this work is to solve these problems. Firstly, the calculation formulae of conditional probabilities for multi-dimensional Markov chain models are proposed under the full independence and conditional independence assumptions. It is noted that multi-dimensional Markov models based on the conditional independence assumption are reasonable because these models avoid the small-class underestimation problem. Then, the methods for determining transition probabilities are given. The vertical transition probabilities are obtained by computing the transition frequencies from drilling data, while the horizontal transition probabilities are estimated by using well data and the elongation ratios according to Walther’s law. Finally, these models are used to simulate the reservoir lithofacies distribution of Tahe oilfield in China. The results show that the conditional independence method performs better than the full independence counterpart in maintaining the true percentage composition and reproducing lithofacies spatial features.

Design and Modeling of Self‐Adapting MAC (SaMAC) Protocol with Inconstant Contention Loss Probabilities

Networks based on IEEE 802.11 standard are one of the main options for deployment in industrial environment. Degradation of throughput in congested networks and short‐term unfairness are well‐known drawbacks of 802.11 DCF and similar MAC protocols. Those shortcomings represent significant limitation in forecasted growth of wireless usage. This is especially important in industrial wireless networks (IWN) where the scalability of wireless MAC is one of the main requirements. In this paper, a novel self‐adapting MAC protocol (SaMAC) is defined and mathematically modeled. SaMAC employs constrained countdown freezing enhanced with shifted window mechanism. As a result, the protocol outperforms 802.11 DCF standard as well as shifted contention window (SCW) and constrained countdown freezing (CPCF) protocols in achieved throughput, fairness, and jitter, while keeping simple implementation. Despite protocol’s simple design, it is shown that its mathematical model is extremely complex. For proposed protocol, the assumption of constant contention loss probability, which is normally used for modeling of MAC schemes, does not hold. In the presented multidimensional Markov chain model, a unique iterative method for determining contention loss probability is developed as well as a method for throughput calculation based on such a chain. Accuracy of the presented model is verified in several network scenarios. Considering the performance of the proposed protocol, authors believe that it could be of benefit to deploy it in heavily loaded wireless networks with timing constraints, such as IWNs.

Space Reduction for a Class of Multidimensional Markov Chains: A Summary and Some Applications

In this paper, we present examples of a class of Markov chains that occur frequently, but whose associated matrices are a challenge to construct efficiently. These are Markov chains that arise as a result of several identical Markov chains running in parallel. Specifically for the cases considered, both the infinitesimal generator matrix for the continuous case, and more so the transition probability matrix for the discrete equivalent, are complex to construct effectively and efficiently. We summarize the algorithms for constructing the associated matrices and present examples of applications, ranging from special queueing problems to reliability issues and order statistics. MATLAB subroutines are provided in an online supplement for the implementation of the algorithms. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0759 .

A Survey on Adaptive Multi-Channel MAC Protocols in VANETs Using Markov Models

A medium access control (MAC) protocol is designed to disseminate safety messages reliably and rapidly to improve the safety and efficiency of vehicles on the road in vehicular ad-hoc networks (VANETs). VANETs, which are created by moving vehicles, have specific properties, such as high node mobility with constrained movements and quick topology changes. Hence, MAC protocols should be designed to adapt to the changing data traffic patterns due to vehicle densities in the VANET environment. The latest multi-channel MAC protocols based on IEEE 802.11p and IEEE 1609.4 standards have higher performance than that of single-channel MAC protocols in every key performance indicator. Specifically, the multi-channel MAC protocols, which adapt themselves to different vehicular traffic densities, can guarantee a bounded transmission delay of real-time safety applications and an increased throughput for non-safety applications. In this paper, we focus on the three following perspectives: First, the multi-channel MAC protocols are studied under saturated and non-saturated data traffic conditions; Second, we study the multi-dimensional Markov chains (up to three dimensions) used in the MAC protocols; and Third, the considered Markov models are compared with real-life application requirements to improve the existing analytical models and protocol designs. Finally, we summarize our findings and discuss the open issues concerning multi-channel MAC protocols for VANETs as a part of the Intelligent Transportation System.

An analytical framework in LEO mobile satellite systems servicing batched Poisson traffic

The authors consider a low earth orbit (LEO) mobile satellite system (MSS) that accepts new and handover calls of multirate service-classes. New calls arrive in the system as batches, following the batched Poisson process. A batch has a generally distributed number of calls. Each call is treated separately from the others and its acceptance is decided according to the availability of the requested number of channels. Handover calls follow also a batched Poisson process. All calls compete for the available channels under the complete sharing policy. By considering the LEO-MSS as a multirate loss system with ‘satellite-fixed’ cells, it can be analysed via a multi-dimensional Markov chain, which yields to a product form solution (PFS) for the steady-state distribution. Based on the PFS, they propose a recursive and yet efficient formula for the determination of the channel occupancy distribution, and consequently, for the calculation of various performance measures including call blocking and handover failure probabilities. The latter are much higher compared to the corresponding probabilities in the case of the classical (and less bursty) Poisson process. Simulation results verify the accuracy of the proposed formulas. Furthermore, they discuss the applicability of the proposed model in software-defined LEO-MSS.