Finite Markov Chain Research Articles

A new stochastic algorithm for proton exchange membrane fuel cell stack design optimization

This paper develops a new stochastic heuristic for proton exchange membrane fuel cell stack design optimization. The problem involves finding the optimal size and configuration of stand-alone, fuel-cell-based power supply systems: the stack is to be configured so that it delivers the maximum power output at the load's operating voltage. The problem apparently looks straightforward but is analytically intractable and computationally hard. No exact solution can be found, nor is it easy to find the exact number of local optima; we, therefore, are forced to settle with approximate or near-optimal solutions. This real-world problem, first reported in Journal of Power Sources 131, poses both engineering challenges and computational challenges and is representative of many of today's open problems in fuel cell design involving a mix of discrete and continuous parameters. The new algorithm is compared against genetic algorithm, simulated annealing, and (1+1)-EA. Statistical tests of significance show that the results produced by our method are better than the best-known solutions for this problem published in the literature. A finite Markov chain analysis of the new algorithm establishes an upper bound on the expected time to find the optimum solution.

Ergodic Transmission Capacity of Wireless Ad Hoc Networks with Interference Management

Most work on wireless network throughput ignores the temporal correlation inherent to wireless channels because it degrades tractability. To better model and quantify the temporal variations of wireless network throughput, this paper introduces a metric termed ergodic transmission capacity (ETC), which includes spatial and temporal ergodicity. All transmitters in the network form a homogeneous Poisson point process and all channels are modeled by a finite state Markov chain. The bounds on outage probability and ETC are characterized, and their scaling behaviors for a sparse and dense network are discussed. From these results, we show that the ETC can be characterized by the inner product of the channel-state related vector and the invariant probability vector of the Markov chain. This indicates that channel-aware opportunistic transmission does not always increase ETC. Finally, we look at outage probability with interference management from a stochastic geometry point of view. The improved bounds on outage probability and ETC due to interference management are characterized and they provide some useful insights on how to effectively manage interference in sparse and dense networks.

State Distributions of Multi-State Consecutive-$k$ Systems

As a flexible, accurate tool for representing practical industrial and military systems, multi-state systems have been paid great attention to in both academic and engineering fields. Multi-state consecutive- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$k$</tex></formula> systems are extensions of binary consecutive- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$k$</tex></formula> systems. In this paper, by using the finite Markov chain imbedding approach, we present a unified formula with the product of matrices for evaluating state distributions of two kinds of multi-state consecutive- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$k$</tex> </formula> systems. Moreover, the analytic expressions are given for the expectation, the probability mass function, and the distribution of the waiting time for the appearance of a state of multi-state consecutive- <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$k$</tex></formula> systems. The formulas obtained in this paper will be useful for start-up demonstration tests, quality control, and so on. Finally, numeral examples are given, which show the flexibility and effectiveness of our model.

The SIS epidemic model with Markovian switching

Population systems are often subject to environmental noise. Motivated by Takeuchi et al. [7], we will discuss in this paper the effect of telegraph noise on the well-known SIS epidemic model. We establish the explicit solution of the stochastic SIS epidemic model, which is useful in performing computer simulations. We also establish the conditions for extinction and persistence for the stochastic SIS epidemic model and compare these with the corresponding conditions for the deterministic SIS epidemic model. We first prove these results for a two-state Markov chain and then generalise them to a finite state space Markov chain. Computer simulations based on the explicit solution and the Euler–Maruyama scheme are performed to illustrate our theory. We include a more realistic example using appropriate parameter values for the spread of Streptococcuspneumoniae in children.

On Markovian solutions to Markov Chain BSDEs

We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows us to connect these equations to coupled systems of ODEs, and hence to give fast numerical methods for the evaluation of Markov-Chain BSDEs.

Stability of a pure random delay system with two-time-scale Markovian switching

This work examines almost sure stability of a pure random delay system whose delay time is modeled by a finite state continuous-time Markov chain with two-time scales. The Markov chain contains a fast-varying part and a slowly-changing part. Using the properties of the weighted occupation measure of the Markov chain, it is shown that the overall systemʼs almost-sure-asymptotic stability can be obtained by using the “averaged” delay. This feature implies that even if some longer delay times may destabilize the system individually, the system may still be stable if their impact is balanced. In other words, the Markov chain becomes a stabilizing factor. Numerical results are provided to demonstrate our results.

Ergodicity Of Fuzzy Markov Chains Based On Simulation Using Halton Sequences

We first introduce fuzzy finite Markov chains and present some of their fundamental properties based on possibility theory. We also bring in a way to convert fuzzy Markov chains to classic Markov chains. In addition, we simulate fuzzy Markov chain using different sizes. It is observed that the most of fuzzy Markov chains not only do have an ergodic behavior, but also they are periodic. Finally, using Halton quasi-random sequence we generate some fuzzy Markov chains which compared to the ones generated by the RAND function of MATLAB. Therefore, we improve the periodicity behavior of fuzzy Markov chains.

Efficient Generation of Random Bits From Finite State Markov Chains

The problem of random number generation from an uncorrelated random source (of unknown probability distribution) dates back to von Neumann's 1951 work. Elias (1972) generalized von Neumann's scheme and showed how to achieve optimal efficiency in unbiased random bits generation. Hence, a natural question is what if the sources are correlated? Both Elias and Samuelson proposed methods for generating unbiased random bits in the case of correlated sources (of unknown probability distribution), specifically, they considered finite Markov chains. However, their proposed methods are not efficient or have implementation difficulties. Blum (1986) devised an algorithm for efficiently generating random bits from degree-2 finite Markov chains in expected linear time, however, his beautiful method is still far from optimality on information-efficiency. In this paper, we generalize Blum's algorithm to arbitrary degree finite Markov chains and combine it with Elias's method for efficient generation of unbiased bits. As a result, we provide the first known algorithm that generates unbiased random bits from an arbitrary finite Markov chain, operates in expected linear time and achieves the information-theoretic upper bound on efficiency.

On the Distribution of the Number of Occurrences of an Order-Preserving Pattern of Length Three in a Random Permutation

Recently, a considerable number of papers in computer science and mathematics examined the number of permutations containing exactly s occurrences of a prescribed order-preserving pattern (or forbidden pattern). It is well known that, mathematically, this is an NP-hard problem. Even in the simple case where the length of an order-preserving pattern is three, the number of permutations of size n containing s (s ≥ 3) order-preserving patterns remains unknown (see Fulmek Adv Appl Math 30(4):607–632, 2003). This manuscript provides a probabilistic approach to enumerate the number of permutations that contain exactly s occurrences of an order-preserving pattern of length three. The method is based on the insertion procedure of the finite Markov chain imbedding technique. Numerical results are provided to illustrate the theoretical results.

Quantifying the uncertainty in change points

Quantifying the uncertainty in the location and nature of change points in time series is important in a variety of applications. Many existing methods for estimation of the number and location of change points fail to capture fully or explicitly the uncertainty regarding these estimates, whilst others require explicit simulation of large vectors of dependent latent variables. This article proposes methodology for approximating the full posterior distribution of various change point characteristics in the presence of parameter uncertainty. The methodology combines recent work on evaluation of exact change point distributions conditional on model parameters via finite Markov chain imbedding in a hidden Markov model setting, and accounting for parameter uncertainty and estimation via Bayesian modelling and sequential Monte Carlo. The combination of the two leads to a flexible and computationally efficient procedure, which does not require estimates of the underlying state sequence. We illustrate that good estimation of the posterior distributions of change point characteristics is provided for simulated data and functional magnetic resonance imaging data. We use the methodology to show that the modelling of relevant physical properties of the scanner can influence detection of change points and their uncertainty.

A fluid introduction to brownian motion &amp; stochastic integration (abstract only)

This presentation provides an introduction to Brownian motion and stochastic integrals using linear fluid flows on finite state Markov chains. Many numerical examples are presented setting the stage for the development of algorithms for stochastic integration via the well-studied and easily understood fluid flow models driven by finite state Markov chains.

Algorithms for Joint Spectrum Allocation and Cooperation Set Partition in Cognitive Radio Networks

The coexistence of multi-primary users and multi-secondary users in cooperative cognitive radio networks motivate the study to propose a joint spectrum allocation and cooperation set partition problem, which so far has not been addressed before. The problem is formulated as a 0-1 integer non-linear programming model. Due to its NP-hardness, the study proposes a suboptimal Centralized Genetic Algorithm (CGA) to show its convergence by modeling it as a homogeneous finite Markov chain. The study then extends CGA to a fully Distributed Genetic Algorithm (DGA) that consists of two phases. The core techniques include a minimum dominate set based cluster partition, a spectrum pre-allocation algorithm in phase 1, and an inter-cluster cooperation set negotiation and cluster fitness refinement algorithm in phase 2. A Fast-Convergent DGA (FDGA) is also devised to reduce the system configuration time. Extensive experiments by simulations demonstrate that in terms of the fitness that reflects the performance of the proposed algorithms: (1) CGA is shown to perform as well as 92% of the optimal solution by brutal search under small network sizes; (2) As the network size increases, due to the massive search space CGA has to deal, DGA and FDGA instead outperform CGA with 20% on average when achieving the same algorithm termination condition; (3) FDGA delivers similar results as DGA while reducing the configuration time significantly, which is more suitable for large-scale networks.

Optimal dividend distribution under Markov regime switching

We investigate the problem of optimal dividend distribution for a company in the presence of regime shifts. We consider a company whose cumulative net revenues evolve as a Brownian motion with positive drift that is modulated by a finite state Markov chain, and model the discount rate as a deterministic function of the current state of the chain. In this setting, the objective of the company is to maximize the expected cumulative discounted dividend payments until the moment of bankruptcy, which is taken to be the first time that the cash reserves (the cumulative net revenues minus cumulative dividend payments) are zero. We show that if the drift is positive in each state, it is optimal to adopt a barrier strategy at certain positive regime-dependent levels, and provide an explicit characterization of the value function as the fixed point of a contraction. In the case that the drift is small and negative in one state, the optimal strategy takes a different form, which we explicitly identify if there are two regimes. We also provide a numerical illustration of the sensitivities of the optimal barriers and the influence of regime switching.

Continuous, Discrete, and Conditional Scan Statistics

The distributions for continuous, discrete, and conditional discrete scan statistics are studied. The approach of finite Markov chain imbedding, which has been applied to random permutations as well as to runs and patterns, is extended to compute the distribution of the conditional discrete scan statistic, defined from a sequence of Bernoulli trials. It is shown that the distribution of the continuous scan statistic induced by a Poisson process defined on (0, 1] is a limiting distribution of weighted distributions of conditional discrete scan statistics. Comparisons of rates of convergence as well as numerical comparisons of various bounds and approximations are provided to illustrate the theoretical results.

Continuous, Discrete, and Conditional Scan Statistics

The distributions for continuous, discrete, and conditional discrete scan statistics are studied. The approach of finite Markov chain imbedding, which has been applied to random permutations as well as to runs and patterns, is extended to compute the distribution of the conditional discrete scan statistic, defined from a sequence of Bernoulli trials. It is shown that the distribution of the continuous scan statistic induced by a Poisson process defined on (0, 1] is a limiting distribution of weighted distributions of conditional discrete scan statistics. Comparisons of rates of convergence as well as numerical comparisons of various bounds and approximations are provided to illustrate the theoretical results.

Markov modulation of a two-sided reflected Brownian motion with application to fluid queues

In this paper, we study a reflected Markov-modulated Brownian motion with a two sided reflection in which the drift, diffusion coefficient and the two boundaries are (jointly) modulated by a finite state space irreducible continuous time Markov chain. The goal is to compute the stationary distribution of this Markov process, which in addition to the complication of having a stochastic boundary can also include jumps at state change epochs of the underlying Markov chain because of the boundary changes. We give the general theory and then specialize to the case where the underlying Markov chain has two states.

Markov Chain Ontology Analysis (MCOA)

BackgroundBiomedical ontologies have become an increasingly critical lens through which researchers analyze the genomic, clinical and bibliographic data that fuels scientific research. Of particular relevance are methods, such as enrichment analysis, that quantify the importance of ontology classes relative to a collection of domain data. Current analytical techniques, however, remain limited in their ability to handle many important types of structural complexity encountered in real biological systems including class overlaps, continuously valued data, inter-instance relationships, non-hierarchical relationships between classes, semantic distance and sparse data.ResultsIn this paper, we describe a methodology called Markov Chain Ontology Analysis (MCOA) and illustrate its use through a MCOA-based enrichment analysis application based on a generative model of gene activation. MCOA models the classes in an ontology, the instances from an associated dataset and all directional inter-class, class-to-instance and inter-instance relationships as a single finite ergodic Markov chain. The adjusted transition probability matrix for this Markov chain enables the calculation of eigenvector values that quantify the importance of each ontology class relative to other classes and the associated data set members. On both controlled Gene Ontology (GO) data sets created with Escherichia coli, Drosophila melanogaster and Homo sapiens annotations and real gene expression data extracted from the Gene Expression Omnibus (GEO), the MCOA enrichment analysis approach provides the best performance of comparable state-of-the-art methods.ConclusionA methodology based on Markov chain models and network analytic metrics can help detect the relevant signal within large, highly interdependent and noisy data sets and, for applications such as enrichment analysis, has been shown to generate superior performance on both real and simulated data relative to existing state-of-the-art approaches.

A Fuzzy Model For Analogical Problem Solving

In this paper we develop a fuzzy model for the description of the process of Analogical Reasoning by representing its main steps as fuzzy subsets of a set of linguistic labels characterizing the individuals' performance in each step and we use the Shannon- Wiener diversity index as a measure of the individuals' abilities in analogical problem solving. This model is compared with a stochastic model presented in author's earlier papers by introducing a finite Markov chain on the steps of the process of Analogical Reasoning. A classroom experiment is also presented to illustrate the use of our results in practice.

Ergodicity Of Fuzzy Markov Chains Based On Simulation Using Halton Sequences

We first introduce fuzzy finite Markov chains and present some of their fundamental properties based on possibility theory. We also bring in a way to convert fuzzy Markov chains to classic Markov chains. In addition, we simulate fuzzy Markov chain using different sizes. It is observed that the most of fuzzy Markov chains not only do have an ergodic behavior, but also they are periodic. Finally, using Halton quasi-random sequence we generate some fuzzy Markov chains which compared to the ones generated by the RAND function of MATLAB. Therefore, we improve the periodicity behavior of fuzzy Markov chains.

OPTIMAL TRAINING POLICY FOR PROMOTION - STOCHASTIC MODELS OF MANPOWER SYSTEMS

In this paper, the optimal planning of manpower training programmes in a manpower system with two grades is discussed. The planning of manpower training within a given organization involves a trade-off between training costs and expected return. These planning problems are examined through models that reflect the random nature of manpower movement in two grades. To be specific, the system consists of two grades, grade 1 and grade 2. Any number of persons in grade 2 can be sent for training and after the completion of training, they will stay in grade 2 and will be given promotion as and when vacancies arise in grade 1. Vacancies arise in grade 1 only by wastage. A person in grade 1 can leave the system with probability p. Vacancies are filled with persons in grade 2 who have completed the training. It is assumed that there is a perfect passing rate and that the sizes of both grades are fixed. Assuming that the planning horizon is finite and is T, the underlying stochastic process is identified as a finite state Markov chain and using dynamic programming, a policy is evolved to determine how many persons should be sent for training at any time k so as to minimize the total expected cost for the entire planning period T.