Markovian Systems Research Articles

Transient analysis of a SIQS model with state capacities using a non-homogeneous Markov system

In this paper, a novel stochastic model is proposed to model the spread of a virus in epidemics. The model is based on a discrete-time non-homogeneous Markov system with state capacities. In order to study the distributions of the state sizes, recursive formulae for their factorial and mixed factorial moments were derived in matrix form. As a consequence, the probability mass function of each state size can be evaluated in the transient period. To avoid the computational complexity of the proposed algorithm, an alternative method for the computation of the state size distributions was recommended. The proposed Markovian approach was then tailored to the characteristics of a SIQS (susceptible-infected-quarantined-susceptible) epidemic scheme, which took into account external infections and the potential for secondary infections. This epidemic model is well-suited for describing infections in computer networks, where the quarantine capacity can be likened to the number of working people (IT professionals) available to restore an infected computer. We presented numerical examples and sensitivity analysis to illustrate the behavior and performance of the system under different scenarios and parameter values. We show that the state capacities and the infection rates have significant effects on the evolution and extinction of the epidemic. We note that the optimal number of employed technicians can be identified, aiming to keep the computer network functional. Higher internal infection rates significantly affect the sustainability of the computer network, while controlling external infections is not always feasible. On the other hand, faster detection rates and higher malware elimination rates will considerably increase the number of computers that remain operational in the long term. Consequently, the quality of services provided by IT technicians plays a crucial role in the system’s viability.

Universal Time-Entanglement Trade-Off in Open Quantum Systems

We demonstrate a surprising connection between pure steady-state entanglement and relaxation time scales in an extremely broad class of Markovian open systems, where two (possibly many-body) systems, A and B, interact locally with a common dissipative environment. This setup also encompasses a broad class of adaptive quantum dynamics based on continuous measurement and feedback. As steady-state entanglement increases, there is generically an emergent strong symmetry that leads to a dynamical slow-down. Using this, we can prove rigorous bounds on relaxation times set by steady-state entanglement. We also find that this time must necessarily diverge for maximal entanglement. To test our bound, we consider the dynamics of a random ensemble of local Lindbladians that support pure steady states, finding that the bound does an excellent job of predicting how the dissipative gap varies with the amount of entanglement. Our work provides general insights into how dynamics and entanglement are connected in open systems and has specific relevance to quantum reservoir engineering. Published by the American Physical Society 2024

Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Markovian System Controlled over a Communication Channel

This paper develops approximation and optimality results for the optimal control of a networked system. In this system, a Markovian process is managed over a finite-rate, noiseless communication channel. Solving this problem involves determining joint optimal coding and control policies that minimize cost over time. While theoretical results on optimal coding and control structures exist, practical implementation has remained largely infeasible for non-linear systems due to the computational complexities and uncountable state spaces. This research introduces an approach that combines structural results with reinforcement learning (RL) techniques. The method uses regularity properties of the system to approximate the uncountable state space with a countable one, which allows reinforcement learning algorithms to achieve near-optimal solutions. Specifically, we establish that finite model approximations (where infinite state spaces are quantized to finite ones) and sliding finite window approximations (where a finite memory "window" of past control actions is maintained at each time step) can be employed to develop near-optimality. These approximations allow the system to be reformulated as a Markov Decision Problem (MDP) with a finite state space, making RL algorithms implementable. The convergence of the reinforcement learning algorithm to a near-optimal policy (under both the previous approximations) is supported by theoretical analysis and performance simulations. We ensure that the solutions obtained are not only computationally feasible but also nearly optimal with respect to the original problem. The applications of this work extend to many networked control systems, especially those with zero-delay coding and partially observable Markov decision processes (POMDPs). By integrating structural results with learning algorithms, this paper provides a practical framework for implementing optimal control in finite-rate environments.

Weak Ergodicity in G-NHMS

In the present we provide the definition of the new concept of the General Non-Homogeneous Markov System (G-NHMS) and establish the expected population structure of a NHMS in the various states. These results will be the basis to build on the new concepts and the basic theorems of what follows.We then establish the set of all possible expected relative distributions of the initial number of memberships at time t and all possible expected relative distributions of the expansion memberships at time t. We call this set the general expected relative population structure in the states of a G-NHMS. We then proceed by providing the new definitions of weak ergodicity in a G-NHMS and weak ergodicity with a geometrical rate of convergence. We then prove the Theorem 4 which is a new building block in the theory of G-NHMS. We also prove a similar theorem under the assumption that relative expansion of the population vanishes at infinity.We then provide a generalization of the coupling theorem for populations. We proceed then to study the asymptotic behavior of a G-NHMS when the input policy consists of independent non-homogeneous Poisson variates for each time interval t-1,t\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\left( t-1,t\\right] $$\\end{document}. It is founded in Theorem 7 that it displays a kind of weak ergodicity behavior, that is, it converges at each step to the row of a stable matrix. This row is independent of the initial distribution and of the asymptotic input policy unlike the results in previous works. Hence it generalizes the result in that works. Finally we illustrate our results numerically for a manpower system with three states.

A comparative study of numerical methods for reliability assessment based on semi-Markov processes

The semi-Markov process, renowned for its versatile applications, has garnered significant attention in recent years. However, deriving closed-form expressions for computing reliability metrics proves challenging when sojourn times deviate from exponential distributions. This paper investigates three numerical techniques for computing the transition function matrix of the semi-Markov process, including the algebraic method, the truncated method, and the iterative method. It delves into their truncation and discretization errors, as well as their computational complexities. Additionally, the Laplace-based method and the semi-Markov-chain-based method are discussed to contrast their effectiveness with the three devised numerical approaches. Building on the linkage between the Markov renewal equation and system reliability metrics, five computational methods are applied to the evaluation of system reliability and availability. A case study on sequential cyber-attacks is presented to illustrate the applicability of these methods, considering sojourn times that follow exponential, gamma, lognormal, and Weibull distributions respectively. The results reveal that the three proposed numerical methods not only achieve high precision and rapid speed but also address scenarios beyond the capability of the Laplace-based method.

Adaptive IMM Smoothing Algorithms for Jumping Markov System With Mismatched Measurement Noise Covariance Matrix

Adaptive IMM Smoothing Algorithms for Jumping Markov System With Mismatched Measurement Noise Covariance Matrix

Finite MX/M/C Queue with State Dependent Service, Two-Class Arrivals and Impatient Customers

Objectives: 1. To compute the transition probabilities for the queueing system to be in different states. 2. To calculate the average system length and mean waiting time under the specified parameters. 3. To investigate and test the impact of system characteristics on the anticipated system length as well as mean waiting times. Method: MATLAB software is used for the Runge-Kutta method to calculate probabilities and system constants. Findings: An increase in Type-I arrival rate λ1 and Type-II arrival rate λ2 will cause both mean waiting times and queue length to increase, and they are reduced with an increase in service rates µ1 and µ2 of Type-I and Type-II classes. A rise in parameters such as special service rate µ3, probability of balking and reneging will result in a reduction in both mean waiting times and queue length. Novelty: This article covers a batch arrival finite Markovian queueing system with multi-server support and consumer impatience. In real time, all arrivals need not enter the queue at the same arrival rates, as it is based on the needs of the customer. In view of this, we have assumed two types of customers along with two different rates of arrival and state-dependent service. We did our work in transient mode, as time can influence the waiting lines. Also determined the system's performance indices and demonstrated the impact of the input parameters on the system's constants. Keywords: Batch Arrival, Customer Impatience, Two Class Customers, State Dependent Service, Multi Server Facility

Long-term accuracy of numerical approximations of SPDEs with the stochastic Navier–Stokes equations as a paradigm

Abstract This work introduces a general framework for establishing the long time accuracy for approximations of Markovian dynamical systems on separable Banach spaces. Our results illuminate the role that a certain uniformity in Wasserstein contraction rates for the approximating dynamics bears on long time accuracy estimates. In particular, our approach yields weak consistency bounds on ${\mathbb{R}}^{+}$ while providing a means to sidestepping a commonly occurring situation where certain higher order moment bounds are unavailable for the approximating dynamics. Additionally, to facilitate the analytical core of our approach, we develop a refinement of certain ‘weak Harris theorems’. This extension expands the scope of applicability of such Wasserstein contraction estimates to a variety of interesting stochastic partial differential equation examples involving weaker dissipation or stronger nonlinearity than would be covered by the existing literature. As a guiding and paradigmatic example, we apply our formalism to the stochastic 2D Navier–Stokes equations and to a semi-implicit in time and spectral Galerkin in space numerical approximation of this system. In the case of a numerical approximation, we establish quantitative estimates on the approximation of invariant measures as well as prove weak consistency on ${\mathbb{R}}^{+}$. To develop these numerical analysis results, we provide a refinement of $L^{2}_{x}$ accuracy bounds in comparison to the existing literature, which are results of independent interest.

Generalized kinetic theory of coarse-grained systems. I. Partial equilibrium and Markov approximations

The general kinetic theory of coarse-grained systems is presented in the abstract formalism of communication theory developed by Shannon and Weaver, Khinchin and Kolmogorov. The martingale theory shows that, under reasonable, general hypotheses, coarse-grained systems can be approximated by generalized Markov systems. For mixing systems, the Kolmogorov entropy production can be defined for nonstationary processes as Kolmogorov defined it for stationary processes.

Stochastic Roll Dynamics of Smooth and Impacting Vessels in Random Waves

Ship instability is frequently associated with extreme roll motion. In this paper smooth and vibro-impact dynamics of vessels rolling under random wave excitations including nonlinearities is investigated. The vessel is represented by a single degree of freedom roll model. The random excitation due to waves and ice is modeled by a sum of random pulses with Poisson arrival rate. Stochastic P-bifurcation analysis is carried out by examining the variation with parameter of the joint probability density functions (jpdf) of the response computed numerically by the solution of the corresponding Fokker–Planck–Kolmogorov (FPK) equation. The finite element (FE), finite difference (FD), path integral (PI) and radial basis neural (RBN) network formulation methods are used in the solution of the FPK equation. The D-bifurcation analysis is analyzed by computing the largest Lyapunov exponent. The mean upcrossing rate is computed using Rice’s formula. The random wave excitation is also modeled as the response of a second order linear filter to white noise and the random response of the ship is investigated by solution of the corresponding FPK equation of the four dimensional Markov system by the FD method. Non-smooth coordinate transformations like the Zhuravlev and Ivanov transformations are used converting the discontinuous systems to equivalent smooth systems in the impact dynamics analysis. The adaptive time stepping method (ATSM) approach is used to accurately locate the discontinuity point and a Brownian tree approach is used to direct the integration along the correct Brownian path. Different models for nonlinear damping and the restoring moment are considered and their effects on ship stochastic response such as chaotic motion, unbounded rotational motion, impact modulation motion and ship capsizing are investigated.

Markov Decision Processes with Observation Costs: Framework and Computation with a Penalty Scheme

We consider Markov decision processes where the state of the chain is only given at chosen observation times and of a cost. Optimal strategies involve the optimization of observation times as well as the subsequent action values. We consider the finite horizon and discounted infinite horizon problems as well as an extension with parameter uncertainty. By including the time elapsed from observations as part of the augmented Markov system, the value function satisfies a system of quasivariational inequalities (QVIs). Such a class of QVIs can be seen as an extension to the interconnected obstacle problem. We prove a comparison principle for this class of QVIs, which implies the uniqueness of solutions to our proposed problem. Penalty methods are then utilized to obtain arbitrarily accurate solutions. Finally, we perform numerical experiments on three applications that illustrate our framework. Funding: J. Tam is supported by the Engineering and Physical Sciences Research Council [Grant 2269738].

Multi-dimensional State Space Collapse in Non-complete Resource Pooling Scenarios

The present paper establishes an explicit multi-dimensional state space collapse (SSC) for parallel-processing systems with arbitrary compatibility constraints between servers and job types. This breaks major new ground beyond the SSC results and queue length asymptotics in the literature which are largely restricted to complete resource pooling (CRP) scenarios where the steady-state queue length vector concentrates around a line in heavy traffic. The multi-dimensional SSC that we establish reveals heavy-traffic behavior which is also far more tractable than the pre-limit queue length distribution, yet exhibits a fundamentally more intricate structure than in the one-dimensional case, providing useful insight into the system dynamics. In particular, we prove that the limiting queue length vector lives in a K-dimensional cone of which the set of spanning vectors is random in general, capturing the delicate interplay between the various job types and servers. For a broad class of systems we provide a further simplification which shows that the collection of random cones constitutes a fixed K-dimensional cone, resulting in a K-dimensional SSC. The dimension~K represents the number of critically loaded subsystems, or equivalently, capacity bottlenecks in heavy-traffic, with K=1 corresponding to conventional CRP scenarios. Our approach leverages probability generating function (PGF) expressions for Markovian systems operating under redundancy policies.

Quantum thermodynamics of boundary time-crystals

Time-translation symmetry breaking is a mechanism for the emergence of non-stationary many-body phases, so-called time-crystals, in Markovian open quantum systems. Dynamical aspects of time-crystals have been extensively explored over the recent years. However, much less is known about their thermodynamic properties, also due to the intrinsic nonequilibrium nature of these phases. Here, we consider the paradigmatic boundary time-crystal system, in a finite-temperature environment, and demonstrate the persistence of the time-crystalline phase at any temperature. Furthermore, we analyze thermodynamic aspects of the model investigating, in particular, heat currents, power exchange and irreversible entropy production. Our work sheds light on the thermodynamic cost of sustaining nonequilibrium time-crystalline phases and provides a framework for characterizing time-crystals as possible resources for, e.g. quantum sensing. Our results may be verified in experiments, for example with trapped ions or superconducting circuits, since we connect thermodynamic quantities with mean value and covariance of collective (magnetization) operators.

Assessment of Reliability of Power Systems Under Adverse Weather Condition Using Markov System Dynamic Method

Abstract We aim to extend the Markov System Dynamic method (MSD) to evaluate the reliability of repairable systems in two-state weather (2SW) models. Increasing the number of components in real practices has limited the use of the Markov method, because of increasing the number of equations and the complexity of the problem. There is no such restriction in the MSD method. Thus, it can be applied to complex systems. Theoretically, we indicate that the 2SW model is a type of linear dynamical system. Therefore, we can use MSD to solve it. Through an example, the validity of the proposed method is confirmed in comparison to the Markov method for the 2SW model. Since the MSD method does not use equations, it can be a preferred alternative for calculating the reliability of 2SW models. Accordingly, we apply the MSD method for assessing the reliability of a four-component system.

Experimental investigation of geometric quantum speed limits in an open quantum system

The quantum speed limit (QSL) is a fundamental lower bound on the evolution time for quantum systems, and its tightness has been observed to be dependent on the properties of the physical process. However, experimental studies exploring the QSL in open quantum systems are still missing. Here, we studied geometric quantum speed limits of a qubit subject to decoherence in an ensemble of chloroform molecules in a Nuclear Magnetic Resonance experiment. We controlled the system-reservoir interaction and the spin relaxation rates by adding a paramagnetic salt, allowing the observation of both Markovian and non-Markovian open system dynamics for the qubit. We used two distinguishability measures of quantum states to assess the speed of the qubit evolution: the quantum Fisher information (QFI) and Wigner-Yanase skew information (WY). For non-Markovianity and low salt concentrations, we found crossovers between QSLs related to those metrics. The WY metric sets the tighter QSL for high concentrations and Markovian dynamics. We also show that QSLs are sensitive even to small fluctuations in spin magnetization.

Performance analysis of a collision channel with abandonments

We consider a Markovian retrial queueing system with customer collisions and abandonment in the context of carrier-sense multiple access systems. Using z-transform techniques, we find a set of first-order differential equations for the probability generating functions of the orbit size when the server is empty, busy, or in the collision phase. We then rely on series expansion techniques to extract approximations for relevant performance measures from this set of differential equations. More precisely, we construct a numerical algorithm to calculate the terms in the series expansions of various factorial moments of the orbit size. To improve the accuracy of our series expansion approach, we apply Wynn’s epsilon algorithm which not only speeds up convergence, but also extends the region of convergence. We illustrate the accuracy of our approach by means of some numerical examples, and find that the method is both fast and accurate for a wide range of the parameter values.

Optimal stopping of conditional McKean–Vlasov jump diffusions

The purpose of this paper is to study the optimal stopping problem of conditional McKean–Vlasov (mean-field) stochastic differential equations with jumps (conditional McKean–Vlasov jump diffusions, for short). We obtain sufficient variational inequalities for a function to be the value function of such a problem and for a stopping time to be optimal.The key is that we combine the conditional McKean–Vlasov equation with the associated stochastic Fokker–Planck partial integro-differential equation for the conditional law of the state. This leads to a Markovian system which can be handled by using a version of a Dynkin formula.Our verification result is illustrated by finding the optimal time to sell in a market with common noise and jumps.

Propagating uncertainty through system dynamics in reproducing kernel Hilbert space

We present a data-driven approach to propagate uncertainty in initial conditions through the dynamics of an unknown system in a reproducing kernel Hilbert space (RKHS). The uncertainty in initial conditions is represented through its kernel mean embedding (KME) in the RKHS. For a discrete-time Markovian dynamical system, we utilize the conditional mean embedding (CME) operator to encode the underlying dynamics. Learning in RKHS often incurs prohibitive data storage requirements. To circumvent said limitation, we propose an algorithm to propagate uncertainty via a learned sparse CME operator, and provide theoretical guarantees on the approximation error for the embedded distribution with time. We empirically study our algorithm over illustrative dynamical systems and power systems.

Universal and nonuniversal probability laws in Markovian open quantum dynamics subject to generalized reset processes.

We consider quantum-jump trajectories of Markovian open quantum systems subject to stochastic in time resets of their state to an initial configuration. The reset events provide a partitioning of quantum trajectories into consecutive time intervals, defining sequences of random variables from the values of a trajectory observable within each of the intervals. For observables related to functions of the quantum state, we show that the probability of certain orderings in the sequences obeys a universal law. This law does not depend on the chosen observable and, in the case of Poissonian reset processes, not even on the details of the dynamics. When considering (discrete) observables associated with the counting of quantum jumps, the probabilities in general lose their universal character. Universality is only recovered in cases when the probability of observing equal outcomes in the same sequence is vanishingly small, which we can achieve in a weak-reset-rate limit. Our results extend previous findings on classical stochastic processes [N. R. Smith et al., Europhys. Lett. 142, 51002 (2023)0295-507510.1209/0295-5075/acd79e] to the quantum domain and to state-dependent reset processes, shedding light on relevant aspects for the emergence of universal probability laws.

New graphical models for sequential data and the improved state estimations by data-conditioned driving noises

A prevalent problem in statistical signal processing, applied statistics, and time series analysis arises from the attempt to identify the hidden state of Markov process based on a set of available noisy observations. In the context of sequential data, filtering refers to the probability distribution of the underlying Markovian system given the measurements made at or before the time of the estimated state. In addition to the filtering, the smoothing distribution is obtained from incorporating measurements made after the time of the estimated state into the filtered solution. This work proposes a number of new filters and smoothers that, in contrast to the traditional schemes, systematically make use of the process noises to give rise to enhanced performances in addressing the state estimation problem. In doing so, our approaches for the resolution are characterized by the application of the graphical models; the graph-based framework not only provides a unified perspective on the existing filters and smoothers but leads us to design new algorithms in a consistent and comprehensible manner. Moreover, the graph models facilitate the implementation of the suggested algorithms through message passing on the graph.