Finite-state Markov Process Research Articles

Langevin dynamics for the probability of finite state Markov processes

Langevin dynamics for the probability of finite state Markov processes

Robustness of reaction–diffusion PDEs predictor-feedback to stochastic delay perturbations

This paper studies the robustness of a PDE backstepping delay-compensated boundary controller for a reaction–diffusion partial differential equation (PDE) with respect to a nominal delay subject to stochastic error disturbance. The stabilization problem under consideration involves random perturbations modeled by a finite-state Markov process that further obstruct the actuation path at the controlled boundary of the infinite-dimension plant. This scenario is useful to describe several actuation failure modes in process control. Employing the recently introduced infinite-dimensional representation of the state of an actuator subject to stochastic input delay for ODEs (Ordinary Differential Equations), we convert the stochastic input delay into r+1 unidirectional advection PDEs, where r corresponds to the number of jump states. Our stability analysis assumes full-state measurement of the spatially distributed plant’s state and relies on a hyperbolic–parabolic PDE-PDE cascade representation of the plant plus actuator dynamics. Integrating the plant and the nominal stabilizing boundary control action, all while considering probabilistic delay disturbances, we establish the proof of mean-square exponential stability as well as the well-posedness of the closed-loop system when random phenomena weaken the nominal actuator compensating effect. Our proof is based on the Lyapunov method, the theory of infinitesimal operator for stability, and C0-semigroup theory for well-posedness. Our stability result refers to the L2-norm of the plant state and the H2-norm of the actuator state. Furthermore, our study presents a qualitative analysis of the maximum deviation of the stochastic disturbance relative to the nominal delay, which is expressed as a function of the severity of the plant instability, namely, the magnitude of the reactivity coefficientand the known upper bound of the transition rate of the underlying stochastic perturbations. Extensive simulation results illustrate the viability of the robustness study.

Improved dynamic event-triggered [formula omitted] control for networked fuzzy singularly perturbed systems with hidden Markov fading channels

In this paper, the dynamic event-triggered H∞ control problem is explored for networked fuzzy slow sampling singularly perturbed systems (S3PSs) subject to hidden Markov fading channels. To alleviate the communication burden more effectively, an improved dynamic event-triggered scheme (DETS) is constructed for networked fuzzy S3PSs, under which two auxiliary internal dynamic variables are considered into event-triggered condition. The fading channel phenomena with randomly time-varying amplitudes are characterized by the finite state Markov process. Meanwhile, considering the actual network mode may be unmeasurable, a hidden Markov mode detector is exploited to estimate the actual network mode, by which the control signal can be adjusted appropriately. By constructing the singular perturbation parameter (SPP)-dependent Lyapunov functional, sufficient conditions are obtained to guarantee the mean square stability of networked fuzzy S3PSs. Besides, the numerical ill-conditioned problem of networked fuzzy S3PSs can be avoided. Finally, a tunnel diode circuit example is proposed to illustrate the effectiveness of the main results.

Managing resource allocation for the recruitment stocking problem

This paper revisits the Recruitment Stocking Problem (RSP), in which a prescribed total number of kits must be allocated across multiple distribution sites, where streams of subjects arrive for kit pickup. A kit can be a test-kit for a new pharmaceutical drug, a medical device, a vaccine, a humanitarian relief survival kit, etc. RSP seeks to minimize the expected recruitment time — the time to recruit a prescribed target number of kits — by identifying an optimal or near-optimal kit allocation across all locations. To this end, we first develop a computational method for evaluating the expected recruitment time using the so-called Randomization Procedure. Specifically, since the recruitment process forms a finite-state continuous-time Markov process, the Randomization Procedure transforms the Markov process with varied transition rates from different states into one with a constant transition rate across all states, and thus substantially simplifies the RSP statistics computation. Further, we develop two search heuristics to efficiently search for optimal or near-optimal kit allocations, especially for large RSP models that are too computationally expensive to optimize by exhaustive search. Finally, we illustrate the efficacy of the proposed heuristics by numerical experiments, and then discuss practical implications and provide managerial insights.

Modeling of Fixed Service Interference in Aeronautical SATCOM Channels

A detailed assessment of the radiofrequency interference from terrestrial fixed service into aeronautical earth stations operating with geostationary satellite orbit networks in the fixed-satellite service is performed. We analyze the impact of the major physical, technical, operational, and regulatory parameters in the interference environment having multiple degrees of freedom to be considered. By performing exhaustive geometric computer simulations, we determine the statistical characteristics of the time-variant radiofrequency interference caused by multiple terrestrial fixed service transmitters using information of existing/registered fixed service stations in France. Subsequently, a stochastic channel model to predict the impact of the terrestrial interference using a higher order finite-state Markov process is developed. Balancing model accuracy and complexity, a reasonable vector quantization clustering algorithm is applied for the partitioning of the interference into representative channel states. For these channel states, we provide a method for scaling this model to different antenna sizes and flight velocities, to be used for development of interference mitigation measures and system design of aeronautical earth stations in motion operating in the 17.7–19.7 GHz frequency range.

A POMDP-Based Antenna Selection for Massive MIMO Communication

We use a partially observable Markov decision process (POMDP) framework to design an optimal antenna selection policy for downlink transmit beamforming at a multi-antenna base station (BS) equipped with only a limited number of RF chains. Assuming that the channel state evolves according to a finite-state Markov process and that only the channel coefficients which correspond to previously selected antennas, are available at the BS, we use the POMDP framework for antenna selection with the aim to maximize the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">long-term expected downlink data rate</i> . To avoid the high computational complexity of the value iteration algorithm, we focus on the myopic policy and prove that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in the case of positively correlated two-state Markov model for the channel over each antenna, the myopic policy is optimal for antenna selection for any number of RF chains</i> . Based on this finding, for general fading channels, we propose to quantize each channel into two levels and apply the myopic policy for antenna selection. Our simulation results show that using this two-state coarse channel quantization for antenna selection results in only a small loss in performance, as compared to the antenna selection technique which uses full channel state information without quantization.

A Fair, Dynamic Load Balanced Task Distribution Strategy for Heterogeneous Cloud Platforms Based on Markov Process Modeling

Load balancing techniques in cloud computing can be applied at three different levels: Virtual machine load balancing, task load balancing, and resource load balancing. At all levels, load balancing should also be implemented in an efficient manner, to increase system performance. In this paper, we propose a fair, in terms of added workload per VM, task load balancing strategy, that aims to improve the average response time and the makespan of the system in the cloud environment. The problem is formulated as an irreducible finite state Markov process, which is known to have a balance equation for each state. From the balance state probabilities we derive the expected utilizations for the virtual machines (VM), which play a vital role in our task allocation approach. In our model, the Load Balancer (LBer) acts as a central server, which uses our proposed fair task allocation scheme to distribute the incoming tasks in a fair, balanced manner among the virtual machines, taking into account their current state as well as their processing capabilities. Our scheme has been compared to recent algorithms that use the particle swarm optimization and the Honey bee foraging scheme to achieve load balancing. Our experimental results show that our proposed scheme outperforms other state of the art schemes in terms of makespan, average response time, and resource utilization and provides lower degree of imbalance.

Dynamic Programs with Shared Resources and Signals: Dynamic Fluid Policies and Asymptotic Optimality

We consider a sequential decision problem involving shared resources and signals in which a decision maker repeatedly observes some exogenous information (the signal), modeled as a finite-state Markov process, then allocates a limited amount of a shared resource across a set of projects. The framework includes a number of applications and generalizes Markovian multi-armed bandit problems by (a) incorporating exogenous information through the signal and (b) allowing for more general resource allocation decisions. Such problems are naturally formulated as stochastic dynamic programs (DPs) but solving the DP is impractical unless the number of projects is small. In this paper, we develop a Lagrangian relaxation and a DP formulation of the corresponding fluid relaxation --- a dynamic fluid relaxation --- that provide upper bounds on the optimal value function as well as a feasible policy. We develop an iterative primal-dual algorithm for solving the dynamic fluid relaxation and analyze the performance of the feasible dynamic fluid policy. Our performance analysis implies, under mild conditions, that the dynamic fluid relaxation bound and feasible policy are asymptotically optimal as the number of projects grows large. Our Lagrangian relaxation uses Lagrange multipliers that depend on the history of past signals in each period: we show that the bounds and analogous policies using restricted forms of Lagrange multipliers (e.g., only depending on the current signal state in each period) in general lead to a performance gap that is linear in the number of projects and thus are not asymptotically optimal in the regime of many projects. We demonstrate the model and results in two applications: (i) a dynamic capital budgeting problem and (ii) a multi-location inventory management problem with limited production capacity and demands that are correlated across locations by a changing market state.

Distributional Properties of Fluid Queues Busy Period and First Passage Times

In this paper, I analyze the distributional properties of the busy period in an on-off fluid queue and the first passage time in a fluid queue driven by a finite state Markov process. In particular, I show that the first passage time has a IFR distribution and the busy period in the Anick-Mitra-Sondhi model has a DFR distribution.

Copula based Measures of Repairable Parallel System with Fault Coverage

The main objective of this study is to analyse the reliability behaviour of parallel systems with three types of failure, namely unit failure, human failure and major failure. For this purpose, we apply three different statistical techniques, namely copula, coverage and copula-coverage. More precisely, this study presents a stochastic model for analysing the behaviour of a multi-state system consisting of two non-identical units by incorporating the concept of coverage factor and two types of repair facilities between failed state to a normal state. The system could be characterized as being in a failed state due to unit failures, human failure and major failures, such as catastrophic and environmental failure. All failure rates are constant and it is assumed that these are exponentially distributed whereas, repair rates follow the Gumbel-Hougaard copula distribution. The entire system is modelled as a finite-state Markov process. Time-dependent reliability measures like availability, reliability and mean time to failure (MTTF) are obtained by supplementary variable techniques and Laplace transformations. The present study provides a comparative analysis for reliability measures among the aforementioned techniques, while a discussion referring to which technique makes the system more reliable is also developed. Furthermore, numerical simulations are presented to validate the analytical results.

Comprehensive Analysis and Optimization of Reliable Viterbi Decoder Circuits Implemented in Modular VLSI Design Logic Styles

The Viterbi Algorithm is a recursive optimal solution for estimating the most likely state sequence of discrete-time finite-state Markov process and Hidden Markov Models (HMM) observed in memoryless noise. The Viterbi algorithm is extensively used for decoding convolutional codes, in the constraint length k that encompasses its use in digital communications specifically in satellite and cellular communications. Storage devices to speed up access, speech synthesis and recognition technologies use the Viterbi algorithm or its variants. In this paper low-power, high-speed and reduced transistor count Viterbi decoding circuits with enhanced error detection capabilities are designed and implemented with signature-based error detection schemes in three design logic styles primarily Conventional CMOS, Hybrid logic and GDI. The significance of the work is the upshot of realizing low latency and low power dissipation with high reliability in the iterative process of finding the least path metric by superseding the subtractor in CSA & PCSA circuits with an optimized comparator. When evaluated against the Traditional/Benchmark CSA & PCSA circuits, the Conventional CMOS design approach attains low power consumption and high accuracy with a reduction in average power dissipation by 4.69% and 3.83% and an improvement in delay performance by 7.89% and 3.79% respectively, with a tradeoff for high area utilization. Whereas, the GDI design approach results in an extreme reduction of transistor count by 71.52% and 74.94% with a weaker logic swing for CSA & PCSA units respectively, complimented by an increase in power dissipation (approximately multiplied by a factor of 5) and deterioration in delay performance by one order of magnitude. The Hybrid logic stages CSA & PCSA units that are 32.52% and 9.27% faster and achieve optimization in area utilization by 48.68% and 51.09% respectively, at the expense of elevated power dissipation by one order of magnitude. All the circuits were designed and simulated using GPDK 90 nm technology libraries on Cadence Design Suite 6.1.6 platform at 27 °C temperature on 1.2 V supply-rail and SPICE codes were generated as well.

Genetic-Algorithm-Assisted Sliding-Mode Control for Networked State-Saturated Systems Over Hidden Markov Fading Channels.

The sliding-mode control (SMC) problem is studied in this article for state-saturated systems over a class of time-varying fading channels. The underlying fading channels, whose channel fading amplitudes (characterized by the expectation and variance) are allowed to be different, are modeled as a finite-state Markov process. A key feature of the problem addressed is to use a hidden Markov mode detector to estimate the actual network mode. The novel model of hidden Markov fading channels (HMFCs) is shown to be more general yet practical than the existing fading channel models. Based on a linear sliding surface, a switching-type SMC law is dedicatedly constructed by just using the estimated network mode. By exploiting the concept of stochastic Lyapunov stability and the approach of hidden Markov models, sufficient conditions are obtained for the resultant SMC systems that ensure both the mean-square stability and the reachability with a sliding region. With the aid of the Hadamard product, a binary genetic algorithm (GA) is developed to solve the proposed SMC design problem subject to some nonconvex constraints induced by the state saturations and the fading channels, where the proposed GA is based on the objective function for optimal reachability. Finally, a numerical example is employed to verify the proposed GA-assisted SMC scheme over the HMFCs.

Markov cubature rules for polynomial processes

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using algebraic techniques. Markov cubature rules aid the tractability of path-dependent tasks such as American option pricing in models where the underlying factors are polynomial processes.

Optimal Control of Markov-Modulated Multiclass Many-Server Queues

We study multiclass many-server queues for which the arrival, service, and abandonment rates are all modulated by a common finite-state Markov process. We assume that the system operates in the “averaged” Halfin–Whitt regime, which means that it is critically loaded in the average sense although not necessarily in each state of the Markov process. We show that, under any static priority policy, the Markov-modulated diffusion-scaled queueing process is exponentially ergodic. This is accomplished by employing a solution to an associated Poisson equation to construct a suitable Lyapunov function. We establish a functional central limit theorem for the diffusion-scaled queueing process and show that the limiting process is a controlled diffusion with piecewise linear drift and constant covariance matrix. We address the infinite-horizon discounted and long-run average (ergodic) optimal control problems and establish asymptotic optimality.

Finite-time H∞ Control of Stochastic Singular Systems with Partly Known Transition Rates via an Optimization Algorithm

This paper is concerned with the problem of finite-time H∞ control for stochastic singular systems with partly known transition rates (TRs). The transition of system parameters follows a finite-state Markov process. Firstly, based on stochastic functional method and linear matrix inequalities (LMIs) technique, sufficient conditions are proposed to ensure finite-time stochastic boundedness (FTSB) and finite-time H∞ stochastic boundedness (FTH∞SB) of considered stochastic singular system. Secondly, by designing a state feedback controller, strict LMI conditions are obtained to guarantee the closed-loop system with partly known TRs to be FTSB and FTH∞SB. Finally, a numerical example is provided to demonstrate the effectiveness of the proposed methods.

Generalized phase-type distributions based on multi-state systems

It is a well-known definition that the time until entering an absorbing state in a finite state Markov process follows a phase-type distribution. In this article we extend this distribution through adding two new events: one is that the number of transitions among states reaches a specified threshold; the other is that the sojourn time in a specified subset of states exceeds a given threshold. The system fails when it enters the absorbing state or two new events happen, whichever occurs first. We develop three models in terms of three circumstances: (i) the two thresholds are constants; (ii) the number of transitions is random while the sojourn time in the specified states is constant; and (iii) the sojourn time in the specified states is random while the number of transitions is constant. To the performance of such systems, we employ the theory of aggregated stochastic processes and obtain closed-form expressions for all reliability indexes, such as point-wise availabilities, various interval availabilities, and distributions of lifetimes. We select special distributions of two thresholds for models 2 and 3, which are the exponential and geometric distributions. The corresponding formulas are presented. Finally, some numerical examples are given to demonstrate the proposed formulas.

Stochastic evolutionary public goods game with first and second order costly punishments in finite populations**Project supported by the National Natural Science Foundation of China (Grant Nos. 71501149 and 71231007), the Soft Science Project of Hubei Province, China (Grant No. 2017ADC122), and the Fundamental Research Funds for the Central Universities, China (Grant No. WUT: 2017VI070).

We study the stochastic evolutionary public goods game with punishment in a finite size population. Two kinds of costly punishments are considered, i.e., first-order punishment in which only the defectors are punished, and second-order punishment in which both the defectors and the cooperators who do not punish the defective behaviors are punished. We focus on the stochastic stable equilibrium of the system. In the population, the evolutionary process of strategies is described as a finite state Markov process. The evolutionary equilibrium of the system and its stochastic stability are analyzed by the limit distribution of the Markov process. By numerical experiments, our findings are as follows. (i) The first-order costly punishment can change the evolutionary dynamics and equilibrium of the public goods game, and it can promote cooperation only when both the intensity of punishment and the return on investment parameters are large enough. (ii) Under the first-order punishment, the further imposition of the second-order punishment cannot change the evolutionary dynamics of the system dramatically, but can only change the probability of the system to select the equilibrium points in the “C+P” states, which refer to the co-existence states of cooperation and punishment. The second-order punishment has limited roles in promoting cooperation, except for some critical combinations of parameters. (iii) When the system chooses “C+P” states with probability one, the increase of the punishment probability under second-order punishment will further increase the proportion of the “P” strategy in the “C+P” states.

Joint densities of hitting times for finite state Markov processes

For a finite state Markov process X and a finite collection {Γ k , k ∈ K} of subsets of its state space, let τ k be the first time the process visits the set Γ k. In general, X may enter some of the Γ k at the same time and therefore the vector τ := (τ k , k ∈ K) may put nonzero mass over lower dimensional regions of R |K| + ; these regions are of the form R s = {t ∈ R |K| + : t i = t j , i, j ∈ s(1)} ∩ |s| l=2 {t : t m < t i = t j , i, j ∈ s(l), m ∈ s(l − 1)} where s is any ordered partition of the set K and s(j) denotes the j th subset of K in the partition s. When |s| < |K|, the density of the law of τ over these regions is said to be singular because it is with respect to the |s|-dimensional Lebesgue measure over the region R s. We derive explicit/recursive and simple to compute formulas for these singular densities and their corresponding tail probabilities over all R s as s ranges over ordered partitions of K. We give a numerical example and indicate the relevance of our results to credit risk modeling. * The research of T.R.

Diffusion approximations for controlled weakly interacting large finite state systems with simultaneous jumps

We consider a rate control problem for an $N$-particle weakly interacting finite state Markov process. The process models the state evolution of a large collection of particles and allows for multiple particles to change state simultaneously. Such models have been proposed for large communication systems (e.g., ad hoc wireless networks) but are also suitable for other settings such as chemical-reaction networks. An associated diffusion control problem is presented and we show that the value function of the $N$-particle controlled system converges to the value function of the limit diffusion control problem as $N\to\infty$. The diffusion coefficient in the limit model is typically degenerate; however, under suitable conditions there is an equivalent formulation in terms of a controlled diffusion with a uniformly nondegenerate diffusion coefficient. Using this equivalence, we show that near optimal continuous feedback controls exist for the diffusion control problem. We then construct near asymptotically optimal control policies for the $N$-particle system based on such continuous feedback controls. Results from some numerical experiments are presented.

Probability‐Weighted Optimal Control for Nonlinear Stochastic Vibrating Systems with Random Time Delay

A probability‐weighted optimal control strategy for nonlinear stochastic vibrating systems with random time delay is proposed. First, by modeling the random delay as a finite state Markov process, the optimal control problem is converted into the one of Markov jump systems with finite mode. Then, upon limiting averaging principle, the optimal control force is approximately expressed as probability‐weighted summation of the control force associated with different modes of the system. Then, by using the stochastic averaging method and the dynamical programming principle, the control force for each mode can be readily obtained. To illustrate the effectiveness of the proposed control, the stochastic optimal control of a two degree‐of‐freedom nonlinear stochastic system with random time delay is worked out as an example.