Nondeterministic Transition Systems Research Articles

Reinforcement Learning for Non-Deterministic Transition Systems With an Application to Symbolic Control

Reinforcement learning (RL) is a well-established framework for the computation of optimal control policies maximizing the expected reward collected along the evolution of Markov decision processes. In this letter, we extend the RL framework to non-deterministic finite transition systems (FTSs), whose solutions are non-unique but not endowed with a probability measure. We show how to dynamically build RL controllers (possibly learning the FTS model just from experience) maximizing the best-case and worst-case return obtained from a trajectory (run) of the model, assuming full-state information. The framework is successfully applied to the case in which the considered transition system is obtained as a finite approximation of a continuous system, also called a symbolic model. Numerical results on the classical mountain car benchmark highlight the potential of the proposed approach.

Memory-loss resilient controller design for temporal logic constraints

ABSTRACT This paper studies the problem of controlling finite nondeterministic transition systems to satisfy constraints given as linear temporal logic properties. A controller architecture is proposed that maps finite fragments of the state trajectory history to control inputs. This approach avoids the standard controller construction that employs an onboard automaton which is fragile to memory loss or errors. In contrast, the proposed architecture requires storing only a finite sequence of previous system states in memory and is therefore resilient to memory loss. In particular, the system will operate unaltered after such a memory-loss event once the system recollects this finite sequence of system states. A generalised algorithm is outlined for controller synthesis in this manner. Additionally, we demonstrate the construction and implementation of such a memory-loss resilient controller through an experimental demonstration on a differential-drive robot that experiences memory-loss events.

Opacity of Nondeterministic Transition Systems: A (Bi)Simulation Relation Approach

In this paper, we propose several opacity-preserving (bi)simulation relations for general nondeterministic transition systems (NTS) in terms of initial-state opacity, current-state opacity, K-step opacity, and infinite-step opacity. We also show how one can leverage quotient construction to compute such relations. In addition, we use a two-way observer method to verify opacity of nondeterministic finite transition systems (NFTSs). As a result, although the verification of opacity for infinite NTSs is generally undecidable, if one can find such an opacity-preserving relation from an infinite NTS to an NFTS, the (lack of) opacity of the NTS can be easily verified over the NFTS which is decidable.

Auditor Product and Controller Synthesis for Nondeterministic Transition Systems With Practical LTL Specifications

Controller design for continuous systems with linear temporal logic (LTL) specifications is a computationally intensive task. Abstracting a discrete transition system from a real-world continuous-state system often results in a state machine with a large number of states and nondeterministic transitions. This makes controller synthesis for LTL specifications difficult specially when the design specification is lengthy. To reduce the complexity, we consider the specifications that are in the conjunctive form of practical LTL patterns. We use auditor product to incrementally restrict the system to satisfy the safety part of each subspecification. The control strategy, that satisfies the liveness part is then calculated by solving a generalized Buchi game on the result of the auditor product of the discrete transition system with all subspecifications. This approach has the same worst case computational complexity as $\text{GR}(1)$ synthesis, but avoids some of the fundamental limitations involved with $Assumption \Rightarrow Guarantee$ formulation of the problem.

Design of Symbolic Controllers for Networked Control Systems

Networked control systems (NCSs) are distributed systems where plants, sensors, actuators, and controllers communicate over shared networks. Nonideal behaviors of the communication network include variable sampling/transmission intervals and communication delays, packet losses, communication constraints, and quantization errors. NCSs have been the object of intensive study in the last few years. However, due to the inherent complexity of NCSs, the current literature focuses on a subset of these nonidealities and mostly considers stability and stabilizability problems. Recent technology advances need different and more complex control objectives to be considered. In this paper, we present first a general model of NCS, including most relevant nonidealities of the communication network; then, we propose a symbolic model approach to the control design with objectives expressed in terms of nondeterministic transition systems. The presented results are based on recent advances in symbolic control design of continuous and hybrid systems. An example in the context of robot motion planning with remote control is included, showing the effectiveness of the proposed approach.

Using progress sets on non-deterministic transition systems for multiple UAV motion planning

This paper presents a new approach for control synthesis of non-deterministic transition systems under Linear Temporal Logic specifications with applications to multiple Unmanned Aerial Vehicles (UAV) motion planning problems. The consideration of such systems is motivated by the non-determinism possibly introduced while abstracting dynamical systems into finite transition systems. More precisely, we consider transition systems enhanced with a progress set describing the fact that the system cannot stay indefinitely in some subset of states. The control synthesis problem is firstly translated into a terminating planning problem. Then, a backward reachability strategy searches for a path from the initial set to the goal set. At each iteration, subsets of states contained in the progress set are added to the path, thus ensuring the reachability to the goal set in finite time. If a solution to the terminating problem is found, the obtained controller is translated back to the initial problem formulation. This approach is validated through an experiment involving two UAVs with a surveillance specification.

Łukasiewicz μ-calculus

The paper explores properties of the Łukasiewicz μ-calculus, or Łμ for short, an extension of Łukasiewicz logic with scalar multiplication and least and greatest fixed-point operators (for monotone formulas). We observe that Łμ terms, with n variables, define monotone piece-wise linear functions fr om [0, 1]n to [0, 1]. Two effective procedures for calculating the output of Łμ terms on rational inputs are presented. We then consider the Łukasiewicz modal μ-calculus, which is obtained by adding box and diamond modalities to Łμ. Alternatively, it can be viewed as a generalization of Kozen’s modal μ-calculus adapted to probabilistic nondeterministic transition systems (PNTS’s). We show how properties expressible in the well-known logic PCTL can be encoded as Łukasiewicz modal μ-calculus formulas. We also show that the algorithms for computing values of Łukasiewicz μ-calculus terms provide automatic (albeit impractical) methods for verifying Łukasiewicz modal μ-calculus properties of finite rational PNTS’s.

Sound statistical model checking for MDP using partial order and confluence reduction

Statistical model checking (SMC) is an analysis method that circumvents the state space explosion problem in model-based verification by combining probabilistic simulation with statistical methods that provide clear error bounds. As a simulation-based technique, it can in general only provide sound results if the underlying model is a stochastic process. In verification, however, models are very often variations of nondeterministic transition systems. In classical exhaustive model checking, partial order reduction and confluence reduction allow the removal of spurious nondeterministic choices. In this paper, we show that both can be adapted to detect and discard such choices on-the-fly during simulation, thus extending the applicability of SMC to a subclass of Markov decision processes. We prove their correctness in a uniform way and study their effectiveness and efficiency using an implementation in the modes simulator for the Modest modelling language. The examples we use highlight the different strengths and limitations of the two approaches. We find that runtime may be affected by unnecessary recomputations, and thus also investigate the feasibility of caching results to speed up simulation at the cost of increased memory usage.

Łukasiewicz mu-Calculus

The paper explores properties of {\L}ukasiewicz mu-calculus, a version of the quantitative/probabilistic modal mu-calculus containing both weak and strong conjunctions and disjunctions from {\L}ukasiewicz (fuzzy) logic. We show that this logic encodes the well-known probabilistic temporal logic PCTL. And we give a model-checking algorithm for computing the rational denotational value of a formula at any state in a finite rational probabilistic nondeterministic transition system.

Automatic Probabilistic Program Verification through Random Variable Abstraction

The weakest pre-expectation calculus has been proved to be a mature theory to analyze quantitative properties of probabilistic and nondeterministic programs. We present an automatic method for proving quantitative linear properties on any denumerable state space using iterative backwards fixed point calculation in the general framework of abstract interpretation. In order to accomplish this task we present the technique of random variable abstraction (RVA) and we also postulate a sufficient condition to achieve exact fixed point computation in the abstract domain. The feasibility of our approach is shown with two examples, one obtaining the expected running time of a probabilistic program, and the other the expected gain of a gambling strategy. Our method works on general guarded probabilistic and nondeterministic transition systems instead of plain pGCL programs, allowing us to easily model a wide range of systems including distributed ones and unstructured programs. We present the operational and weakest precondition semantics for this programs and prove its equivalence.

Model checking transactional memories

Model checking transactional memories (TMs) is difficult because of the unbounded number, length, and delay of concurrent transactions, as well as the unbounded size of the memory. We show that, under certain conditions satisfied by most TMs we know of, the model checking problem can be reduced to a finite-state problem, and we illustrate the use of the method by proving the correctness of several TMs, including two-phase locking, DSTM, and TL2. The safety properties we consider include strict serializability and opacity; the liveness properties include obstruction freedom, livelock freedom, and wait freedom. Our main contribution lies in the structure of the proofs, which are largely automated and not restricted to the TMs mentioned above. In a first step we show that every TM that enjoys certain structural properties either violates a requirement on some program with two threads and two shared variables, or satisfies the requirement on all programs. In the second step, we use a model checker to prove the requirement for the TM applied to a most general program with two threads and two variables. In the safety case, the model checker checks language inclusion between two finite-state transition systems, a nondeterministic transition system representing the given TM applied to a most general program, and a deterministic transition system representing a most liberal safe TM applied to the same program. The given TM transition system is nondeterministic because a TM can be used with different contention managers, which resolve conflicts differently. In the liveness case, the model checker analyzes fairness conditions on the given TM transition system.

Discrete state estimators for systems on a lattice

We address the problem of estimating discrete variables in a class of deterministic transition systems in which the continuous variables are available for measurement. We propose a novel approach to the estimation of discrete variables using lattice theory that overcomes some of the severe complexity issues encountered in previous work. The methodology proposed for the estimation of discrete variables is general as it is applicable to any observable system. Extensions generalize the approach to nondeterministic transition systems. The proposed estimator is finally constructed for a multi-robot system involving two teams competing against each other.

Abstract interpretation of programs as Markov decision processes

We propose a formal language for the specification of trace properties of probabilistic, nondeterministic transition systems, encompassing the properties expressible in Linear Time Logic. Those formulas are in general undecidable on infinite deterministic transition systems and thus on infinite Markov decision processes. This language has both a semantics in terms of sets of traces, as well as another semantics in terms of measurable functions; we give and prove theorems linking the two semantics. We then apply abstract interpretation-based techniques to give upper bounds on the worst-case probability of the studied property. We propose an enhancement of this technique when the state space is partitioned — for instance along the program points — allowing the use of faster iteration methods.

A coalgebraic introduction to CSP

The paper presents a first step of a coalgebraic analysis of the concept of communicating sequential processes as introduced by Hoare in [3]. We make apparent the strong relationship between CSP and partial automata, i.e., special coalgebras. Thereby it turns out that [3] is only dealt with very special automata, namely, with final automata, i.e., automata where the difference between the concepts of state and of process, respectively, disappears. The coalgebraic approach will allow us to develop a proper model theory for process calculi. As first steps in this direction we outline firstly how operations on processes can be generalized in a compatible way to constructions on automata. Secondly, we present a new method for solving recursive process equations. Finally, we discuss that the nondeterminism in CSP can not be modeled based on nondeterministic transition systems usually considered in the coalgebraic literature [5].

Linear dynamic Kahn networks are deterministic

The (first part of the) Kahn principle states that networks with deterministic nodes are deterministic on the I/O level: for each network, different executions provided with the same input streams deliver the same output streams. The Kahn principle has thus far not been proved for dynamic, nondeterministic networks. We consider a simple language L containing the fork-statement. For this language we introduce a nondeterministic transition system which defines all interleavings consisting of basic steps, for all possible executions of a program. We prove that, although on the execution level there is much nondeterminism, this nondeterminism disappears because all executions deliver the same output stream (or a prefix of it), given the same input stream. This proves the Kahn principle for linear, nondeterministic dynamic networks.

“A la Burstall” intermittent assertions induction principles for proving inevitability properties of programs

We formalize Burstall's (1974) intermittent assertions method (initially conceived for proving total correctness of sequential programs) and generalize it to handle inevitability proofs for nondeterministic transition systems, hence (in particular) parallel programs. Programs are modeled by transition systems, program executions by sets of complete traces and program properties by inevitability properties of transition systems (generalizing total correctness of programs). It follows that the study is independent of any particular programming language. The basic proof principle that we derive from Burstall's and Manna and Waldinger's (1978) description of the intermittent assetions method is shown to be sound. It is also semantically complete under a condition on execution traces and inevitable properties. This condition is satisfied when considering inevitability properties such as total correctness or properties involving unary assertions on states. However, we conjecture that (even for deterministic programs) the basic proof principle is not complete when considering arbitrary binary inevitability properties (which relate state values at different “time instants”). This conjecture leads us to a generalization of Burstall's intermittent assertions method using transfinite induction (to handle unbounded nondeterminism) and using auxiliary or ghost variables in the limited form of ternary intermittent assertions (which can relate state values on program entry and at two other different “time instants”). From this generalized proof principle we derive a series of induction principles so as to broaden the allowable forms of proofs. Also we obtain more abstract and hence better understood formalizations of Burstall's method. All proof principles are sound and semantically complete (essentially, as noticed by Manna and Waldinger, because Burstall's method can be used to express “à la Floyd” proofs). However, we prove a stronger semantic completeness result in the sense that the propositions and lemmas involved in “à la Burstall” inevitability proofs can be chosen freely (at least under necessary and sufficient conditions that we specify accurately).

Concurrent transition systems

Concurrent transition systems (CTS's), are ordinary nondeterministic transition systems that have been equipped with additional concurrency information. This concurrency information is specified in terms of a binary residual operation on transitions, which describes how certain pairs of transitions “commute.” The defining axioms for a CTS generate a rich algebraic theory, which we develop in detail. Each CTS C freely generates a complete CTS or computational category C ∗, whose arrows are equivalence classes of finite computation sequences, modulo a congruence induced by the residual operation. The notion “computation tree” for ordinary transition systems generalizes to computation diagram for CTS's, leading to the convenient definition of computations of a CTS as the ideals of its computation diagram. A pleasant property of this definition is that the notion of a maximal ideal in certain circumstances can serve as a replacement for the more troublesome notion of a fair computation sequence To illustrate the utility of CTS's, we use them to define and investigate a dataflow-like model of concurrent computation. The model consists of machines, which generalize the sequential machines of classical automata theory, and various operations (parallel product, input and output relabeling, and feedback) on machines that correspond to ways of combining machines into networks. Using our definition of computations as ideals, we define a natural notion of observable equivalence of machines, and show that it is the largest congruence, respecting parallel product and feedback, that does not relate two machines with distinct input/output relations. In an attempt to obtain information about the algebra of observable equivalence classes, we investigate a series of abstractions of the machine model, show that these abstractions respect the feedback operation, and characterize the homomorphic image of this operation in each case. A byproduct of our analysis is a structural characterization of a large class of processes with functional input/output behavior, and a proof that the feedback operation on such processes obeys Kahn's fixed-point principle.

Sometime = always + recursion ? always on the equivalence of the intermittent and invariant assertions methods for proving inevitability properties of programs

We propose and compare two induction principles called “always” and “sometime” for proving inevitability properties of programs. They are respective formalizations and generalizations of Floyd invariant assertions and Burstall intermittent assertions methods for proving total correctness of sequential programs whose methodological advantages or disadvantages have been discussed in a number of previous papers. Both principles are formalized in the abstract setting of arbitrary nondeterministic transition systems and illustrated by appropriate examples. The “sometime” method is interpreted as a recursive application of the “always” method. Hence “always” can be considered as a special case of “sometime”. These proof methods are strongly equivalent in the sense that a proof by one induction principle can be rewritten into a proof by the other one. The first two theorems of the paper show that an invariant for the “always” method can be translated into an invariant for the “sometime” method even if every recursive application of the later is required to be of finite length. The third and main theorem of the paper shows how to translate an invariant for the “sometime” method into an invariant for the “always” method. It is emphasized that this translation technique follows the idea of transforming recursive programs into iterative ones. Of course, a general translation technique does not imply that the original “sometime” invariant and the resulting “always” invariant are equally understandable. This is illustrated by an example.