Infinite State Space Research Articles

Analysis of a Semi-Open Queuing Network with a State Dependent Marked Markovian Arrival Process, Customers Retrials and Impatience

We consider a queuing network with single-server nodes and heterogeneous customers. The number of customers, which can obtain service simultaneously, is restricted. Customers that cannot be admitted to the network upon arrival make repeated attempts to obtain service. The service time at the nodes is exponentially distributed. After service completion at a node, the serviced customer can transit to another node or leave the network forever. The main features of the model are the mutual dependence of processes of customer arrivals and retrials and the impatience and non-persistence of customers. Dynamics of the network are described by a multidimensional Markov chain with infinite state space, state inhomogeneous behavior and special structure of the infinitesimal generator. The explicit form of the generator is derived. An effective algorithm for computing the stationary distribution of this chain is recommended. The expressions for computation of the key performance measures of the network are given. Numerical results illustrating the importance of the account of the mentioned features of the model are presented. The model can be useful for capacity planning, performance evaluation and optimization of various wireless telecommunication networks, transportation and manufacturing systems.

Local approximation of Markov chains in time and space

ABSTRACTIn epidemic modelling, the emergence of a disease is characterized by the low numbers of infectious individuals. Environmental randomness impacts outcomes such as the outbreak or extinction of the disease in this case. This randomness can be accounted for by modelling the system as a continuous time Markov chain, . The probability of extinction given some initial state is the probability of hitting a subset of the state space associated with extinction for the initial state. This hitting probability can be studied by passing to the discrete time Markov chain (DTMC), . An approach is presented to approximate a DTMC on a countably infinite state space by a DTMC on a finite state space for the purpose of solving general hitting problems. This approach is applied to approximate the probability of disease extinction in an epidemic model. It is also applied to evaluate a heterogeneous disease control strategy in a metapopulation.

Bounded model checking of signal temporal logic properties using syntactic separation

Signal temporal logic (STL) is a temporal logic formalism for specifying properties of continuous signals. STL is widely used for analyzing programs in cyber-physical systems (CPS) that interact with physical entities. However, existing methods for analyzing STL properties are incomplete even for bounded signals, and thus cannot guarantee the correctness of CPS programs. This paper presents a new symbolic model checking algorithm for CPS programs that is refutationally complete for general STL properties of bounded signals. To address the difficulties of dealing with an infinite state space over a continuous time domain, we first propose a syntactic separation of STL, which decomposes an STL formula into an equivalent formula so that each subformula depends only on one of the disjoint segments of a signal. Using the syntactic separation, an STL model checking problem can be reduced to the satisfiability of a first-order logic formula, which is decidable for CPS programs with polynomial dynamics using satisfiability modulo theories (SMT). Unlike the previous methods, our method can verify the correctness of CPS programs for STL properties up to given bounds.

Trace abstraction modulo probability

We propose trace abstraction modulo probability, a proof technique for verifying high-probability accuracy guarantees of probabilistic programs. Our proofs overapproximate the set of program traces using failure automata, finite-state automata that upper bound the probability of failing to satisfy a target specification. We automate proof construction by reducing probabilistic reasoning to logical reasoning: we use program synthesis methods to select axioms for sampling instructions, and then apply Craig interpolation to prove that traces fail the target specification with only a small probability. Our method handles programs with unknown inputs, parameterized distributions, infinite state spaces, and parameterized specifications. We evaluate our technique on a range of randomized algorithms drawn from the differential privacy literature and beyond. To our knowledge, our approach is the first to automatically establish accuracy properties of these algorithms.

Cooperative Spectrum Sensing in Multichannel Cognitive Radio Networks With Energy Harvesting

This paper investigates cooperative spectrum sensing in multi-channel cognitive radio networks (CRNs) with energy harvesting. Our goal is two-fold: first, to determine the optimal sensing parameters for effective management of the limited energy budget in order to maximize the achievable throughput, and second, to exploit the benefits of a practical CRN towards improving the performance of the energy constrained CRN. Two different scenarios are considered. In the first, the secondary user (SU) is assigned a single radio frequency (RF) harvesting source, while in the second, the SU is assigned multiple RF harvesting sources and can opportunistically harvest from any of the sources. For these scenarios, the problem is formulated as a stochastic optimal control system with infinite and continuous state and action spaces. This is known to be computationally intractable and becomes even more complicated in a two-dimensional problem such as considered. In order to reduce the computational complexity, a myopic optimization approach is taken, and the problem is formulated into a mixed integer nonlinear problem (MINLP) to determine the channel assignment, the sensing duration, the distribution of the sensing duration associated with the assigned channels and the detection threshold under the constraint of energy causality and primary user (PU) protection. A near-optimal solution is obtained to the MINLP based on the alternating convex optimization technique. The simulation results obtained show that the considered work can improve the amount of energy harvested and, by extension, the active probability of the SUs by exploiting the multi-channel benefits of practical CRN for enhanced throughput.

The Myopic Stable Set for Social Environments

We introduce a new solution concept for models of coalition formation, called the myopic stable set. The myopic stable set is defined for a very general class of social environments and allows for an infinite state space. We show that the myopic stable set exists and is non-empty. Under minor continuity conditions, we also demonstrate uniqueness. Furthermore, the myopic stable set is a superset of the core and of the set of pure strategy Nash equilibria in noncooperative games. Additionally, the myopic stable set generalizes and unifies various results from more specific environments. In particular, the myopic stable set coincides with the coalition structure core in coalition function form games if the coalition structure core is non-empty; with the set of stable matchings in the standard one-to-one matching model; with the set of pairwise stable networks and closed cycles in models of network formation; and with the set of pure strategy Nash equilibria in finite supermodular games, finite potential games, and aggregative games. We illustrate the versatility of our concept by characterizing the myopic stable set in a model of Bertrand competition with asymmetric costs, for which the literature so far has not been able to fully characterize the set of all (mixed) Nash equilibria.

Why is Kemeny’s constant a constant?

Abstract In their 1960 book on finite Markov chains, Kemeny and Snell established that a certain sum is invariant. The value of this sum has become known as Kemeny’s constant. Various proofs have been given over time, some more technical than others. We give here a very simple physical justification, which extends without a hitch to continuous-time Markov chains on a finite state space. For Markov chains with denumerably infinite state space, the constant may be infinite and even if it is finite, there is no guarantee that the physical argument will hold. We show that the physical interpretation does go through for the special case of a birth-and-death process with a finite value of Kemeny’s constant.

Proper Policies in Infinite-State Stochastic Shortest Path Problems

We consider stochastic shortest path problems with infinite state and control spaces, a nonnegative cost per stage, and a termination state. We extend the notion of a proper policy, a policy that terminates within a finite expected number of steps, from the context of finite state space to the context of infinite state space. We consider the optimal cost function $J^*$ , and the optimal cost function $\hat{J}$ over just the proper policies. We show that $J^*$ and $\hat{J}$ are the smallest and largest solutions of Bellman's equation, respectively, within a suitable class of Lyapounov-like functions. If the cost per stage is bounded, these functions are those that are bounded over the effective domain of $\hat{J}$ . The standard value iteration algorithm may be attracted to either $J^*$ or $\hat{J}$ , depending on the initial condition.

Market Model Benchmark Suite for Machine Learning Techniques

Recent developments in deep-reinforcement learning have yielded promising results in artificial games and test domains. To explore opportunities and evaluate the performance of these machine learning techniques, various benchmark suites are available, such as the Arcade Learning Environment, rllab, OpenAI Gym, and the StarCraft II Learning Environment. This set of benchmark suites is extended with the open business simulation model described here, which helps to promote the use of machine learning techniques as valueadding tools in the context of strategic decision making and economic model calibration and harmonization. The benchmark suite extends the current state-of-the-art problems for deep-reinforcement learning by offering an infinite state and action space for multiple players in a non-zero-sum game environment of imperfect information. It provides a model that can be characterized as both a credit assignment problem and an optimization problem. Experiments with this suite?s deep-reinforcement learning algorithms, which yield remarkable results for various artificial games, highlight that stylized market behavior can be replicated, but the infinite action space, simultaneous decision making, and imperfect information pose a computational challenge. With the directions provided, the benchmark suite can be used to explore new solutions in machine learning for strategic decision making and model calibration.

An Abstraction-Refinement Methodologyfor Reasoning about Network Games†

Network games (NGs) are played on directed graphs and are extensively used in network design and analysis. Search problems for NGs include finding special strategy profiles such as a Nash equilibrium and a globally-optimal solution. The networks modeled by NGs may be huge. In formal verification, abstraction has proven to be an extremely effective technique for reasoning about systems with big and even infinite state spaces. We describe an abstraction-refinement methodology for reasoning about NGs. Our methodology is based on an abstraction function that maps the state space of an NG to a much smaller state space. We search for a global optimum and a Nash equilibrium by reasoning on an under- and an over-approximation defined on top of this smaller state space. When the approximations are too coarse to find such profiles, we refine the abstraction function. We extend the abstraction-refinement methodology to labeled networks, where the objectives of the players are regular languages. Our experimental results demonstrate the effectiveness of the methodology.

Approximate adaptive uniformization of continuous-time Markov chains

• Continuous-time Markov chains on large, possibly infinite state spaces. • Construction of a computationally tractable approximating process/model. • Computable upper and lower bounds for transient state probabilities. • Different strategies for applying approximate adaptive uniformization. We consider the approximation of transient (time dependent) probability distributions of discrete-state continuous-time Markov chains on large, possibly infinite state spaces. A framework for approximate adaptive uniformization is provided, which generalizes the well-known uniformization technique and many of its variants. Based on a birth process and a discrete-time Markov chain a computationally tractable approximating process/model is constructed. We investigate the theoretical properties of this process and prove that it yields computable lower and upper bounds for the desired transient probabilities. Finally, we discuss different specific ways of performing approximate adaptive uniformization and analyze the corresponding approximation errors. The application is illustrated by an example of a stochastic epidemic model.

Error bounds for augmented truncation approximations of continuous-time Markov chains

This paper considers the augmented truncation approximation of the generator of an ergodic continuous-time Markov chain with a countably infinite state space. The main purpose of this paper is to present bounds for the absolute difference between the stationary distributions of the original generator and its augmented truncation. As examples, we apply the bounds to an M∕M∕s retrial queue and an upper Hessenberg Markov chain.

On compact vector formats in the solution of the chemical master equation with backward differentiation

SummaryA stochastic chemical system with multiple types of molecules interacting through reaction channels can be modeled as a continuous‐time Markov chain with a countably infinite multidimensional state space. Starting from an initial probability distribution, the time evolution of the probability distribution associated with this continuous‐time Markov chain is described by a system of ordinary differential equations, known as the chemical master equation (CME). This paper shows how one can solve the CME using backward differentiation. In doing this, a novel approach to truncate the state space at each time step using a prediction vector is proposed. The infinitesimal generator matrix associated with the truncated state space is represented compactly, and exactly, using a sum of Kronecker products of matrices associated with molecules. This exact representation is already compact and does not require a low‐rank approximation in the hierarchical Tucker decomposition (HTD) format. During transient analysis, compact solution vectors in HTD format are employed with the exact, compact, and truncated generated matrices in Kronecker form, and the linear systems are solved with the Jacobi method using fixed or adaptive rank control strategies on the compact vectors. Results of simulation on benchmark models are compared with those of the proposed solver and another version, which works with compact vectors and highly accurate low‐rank approximations of the truncated generator matrices in quantized tensor train format and solves the linear systems with the density matrix renormalization group method. Results indicate that there is a reason to solve the CME numerically, and adaptive rank control strategies on compact vectors in HTD format improve time and memory requirements significantly.

Combinatorial Lévy processes

Combinatorial Lévy processes evolve on general state spaces of combinatorial structures, of which standard examples include processes on sets, graphs and $n$-ary relations and more general possibilities are given by processes on graphs with community structure and multilayer networks. In this setting, the usual Lévy process properties of stationary, independent increments are defined in an unconventional way in terms of the symmetric difference operation on sets. The main theorems characterize both finite and infinite state space combinatorial Lévy processes by a unique $\sigma$-finite measure. Under the additional assumption of exchangeability, I prove a more explicit characterization by which every exchangeable combinatorial Lévy process corresponds to a Poisson point process on the same state space. Associated behavior of the projection into a space of limiting objects reflects certain structural features of the covering process.

Robust Control of a Cable From a Hyperbolic Partial Differential Equation Model

This article presents a detailed study of the robust control of a cable’s vibrations, with emphasis on considering a model of infinite dimension. Indeed, using a partial differential equation model of the vibrations of an inclined cable with sag, we are interested in studying the application of $\mathcal H_{\infty }$ -robust feedback control to this infinite dimensional system. The approach relies on Riccati equations to stabilize the system under measurement feedback when it is subjected to external disturbances. Henceforth, this article focuses on the construction of a standard linear infinite dimensional state space description of the cable under consideration before writing its approximation of finite dimension and studying the $\mathcal H_{\infty }$ feedback control of vibrations with partial observation of the state in both cases. The closed-loop system is numerically simulated to illustrate the effectiveness of the resulting control law.

Computational Identification of Irreducible State-Spaces for Stochastic Reaction Networks

Stochastic models of reaction networks are becoming increasingly important in systems biology. In these models, the dynamics is generally represented by a continuous-time Markov chain whose states denote the copy-numbers of the constituent species. The state-space on which this process resides is a subset of a nonnegative integer lattice, and for many examples of interest, this state-space is countably infinite. This causes numerous problems in analyzing the Markov chain and understanding its long-term behavior. These problems are further confounded by the presence of conservation relations among species which constrain the dynamics in complicated ways. In this paper we provide a linear-algebraic procedure to disentangle these conservation relations and represent the state-space in a special decomposed form, based on the copy-number ranges of various species and dependencies among them. This decomposed form is advantageous for analyzing the stochastic model, and for a large class of networks we demonstrate how this form can be used for finding all the closed communication classes for the Markov chain within the infinite state-space. Such communication classes are irreducible state-spaces for the dynamics and they support all the extremal stationary distributions for the Markov chain. Hence our results provide important insights into the long-term behavior and stability properties of stochastic models of reaction networks. We discuss how the knowledge of these irreducible state-spaces can be used in many ways such as speeding-up stochastic simulations of multiscale networks or in identifying the stationary distributions of complex-balanced networks. We illustrate our results with several examples of gene-expression networks from systems biology.

Control Space Reduction and Real-Time Accurate Modeling of Continuum Manipulators Using Ritz and Ritz–Galerkin Methods

To address the challenges with real-time accurate modeling of multisegment continuum manipulators in the presence of significant external and body loads, we introduce a novel series solution for variable-curvature Cosserat rod static and Lagrangian dynamic methods. By combining a modified Lagrange polynomial series solution, based on experimental observations, with Ritz and Ritz–Galerkin methods, the infinite modeling state space of a continuum manipulator is minimized to geometrical position of a handful of physical points (in our case two). As a result, a unified easy to implement vector formalism is proposed for the nonlinear impedance and configuration control. We showed that by considering the mechanical effects of highly elastic axial deformation, the model accuracy is increased up to 6%. The proposed model predicts experimental results with 6%–8% (4–6 mm) mean error for the Ritz–Galerkin method in static cases and 16%–20% (12–14 mm) mean error for the Ritz method in dynamic cases, in planar and general three-dimensional motions. Comparing to five different models in the literature, our approximate solution is shown to be more accurate with the smallest possible number of modeling states and suitable for real-time modeling, observation, and control applications.

The Evolutionary Game of Pressure (or Interference), Resistance and Collaboration

In this paper we extend the framework of the evolutionary inspection game put forward recently by the author and coworkers to a large class of conflict interactions to address the pressure executed by the major player (or principal) on the large group of small players who can resist this pressure or collaborate with the major player. We prove rigorous results on the convergence of various Markov decision models of interacting small agents (including evolutionary growth), i.e., pairwise, in groups and by coalition formation, to a deterministic evolution on the distributions of the state spaces of small players paying main attention to situations with an infinite state-space of small players. We supply precise rates of convergence. The theoretical results of the paper are applied to the analysis of the processes of inspection, corruption, cyber-security, counter-terrorism, banks and firms merging, strategically enhanced preferential attachment, and many other.

Verifying chemical reaction network implementations: A pathway decomposition approach

The emerging fields of genetic engineering, synthetic biology, DNA computing, DNA nanotechnology, and molecular programming herald the birth of a new information technology that acquires information by directly sensing molecules within a chemical environment, stores information in molecules such as DNA, RNA, and proteins, processes that information by means of chemical and biochemical transformations, and uses that information to direct the manipulation of matter at the nanometer scale. To scale up beyond current proof-of-principle demonstrations, new methods for managing the complexity of designed molecular systems will need to be developed. Here we focus on the challenge of verifying the correctness of molecular implementations of abstract chemical reaction networks, where operation in a well-mixed “soup” of molecules is stochastic, asynchronous, concurrent, and often involves multiple intermediate steps in the implementation, parallel pathways, and side reactions. This problem relates to the verification of Petri nets, but existing approaches are not sufficient for providing a single guarantee covering an infinite set of possible initial states (molecule counts) and an infinite state space potentially explored by the system given any initial state. We address these issues by formulating a new theory of pathway decomposition that provides an elegant formal basis for comparing chemical reaction network implementations, and we present an algorithm that computes this basis. Our theory naturally handles certain situations that commonly arise in molecular implementations, such as what we call “delayed choice,” that are not easily accommodated by other approaches. We further show how pathway decomposition can be combined with weak bisimulation to handle a wider class that includes most currently known enzyme-free DNA implementation techniques. We anticipate that our notion of logical equivalence between chemical reaction network implementations will be valuable for other molecular implementations such as biochemical enzyme systems, and perhaps even more broadly in concurrency theory.

Simplification of Markov chains with infinite state space and the mathematical theory of random gene expression bursts.

Here we develop an effective approach to simplify two-time-scale Markov chains with infinite state spaces by removal of states with fast leaving rates, which improves the simplification method of finite Markov chains. We introduce the concept of fast transition paths and show that the effective transitions of the reduced chain can be represented as the superposition of the direct transitions and the indirect transitions via all the fast transition paths. Furthermore, we apply our simplification approach to the standard Markov model of single-cell stochastic gene expression and provide a mathematical theory of random gene expression bursts. We give the precise mathematical conditions for the bursting kinetics of both mRNAs and proteins. It turns out that random bursts exactly correspond to the fast transition paths of the Markov model. This helps us gain a better understanding of the physics behind the bursting kinetics as an emergent behavior from the fundamental multiscale biochemical reaction kinetics of stochastic gene expression.