Reachability Specifications Research Articles

IntervalMDP.jl: Accelerated Value Iteration for Interval Markov Decision Processes

In this paper, we present IntervalMDP.jl, a Julia package for probabilistic analysis of interval Markov Decision Processes (IMDPs). IntervalMDP.jl facilitates the synthesis of optimal strategies and verification of IMDPs against reachability specifications and discounted reward properties. The library supports sparse matrices and is compatible with data formats from common tools for the analysis of probabilistic models, such as PRISM. A key feature of IntervalMDP.jl is that it presents both a multi-threaded CPU and a GPU-accelerated implementation of value iteration algorithms for IMDPs. In particular, IntervalMDP.jl takes advantage of the Julia type system and the inherently parallelizable nature of value iteration to improve the efficiency of performing analysis of IMDPs. On a set of examples, we show that IntervalMDP.jl substantially outperforms existing tools for verification and strategy synthesis for IMDPs in both computation time and memory consumption.

Data-Driven Abstractions via Binary-Tree Gaussian Processes for Formal Verification

To advance formal verification of stochastic systems against temporal logic requirements for handling unknown dynamics, researchers have been designing data-driven approaches inspired by breakthroughs in the underlying machine learning techniques. As one promising research direction, abstraction-based solutions based on Gaussian process (GP) regression have become popular for their ability to learn a representation of the latent system from data with a quantified error. Results obtained based on this model are then translated to the true system via various methods. In a recent publication, GPs using a so-called binary-tree kernel have demonstrated a polynomial speedup w.r.t. the size of the data compared to their vanilla version, outcompeting all existing sparse GP approximations. Incidentally, the resulting binary-tree Gaussian process (BTGP) is characteristic for its piecewise-constant posterior mean and covariance functions, naturally abstracting the input space into discrete partitions. In this paper, we leverage this natural abstraction of the BTGP for formal verification, eliminating the need for cumbersome abstraction and error quantification procedures. We show that the BTGP allows us to construct an interval Markov chain model of the unknown system with a speedup that is polynomial w.r.t. the size of the abstraction compared to alternative approaches. We provide a delocalized error quantification via a unified formula even when the true dynamics do not live in the function space of the BTGP. This allows us to compute upper and lower bounds on the probability of satisfying reachability specifications that are robust to both aleatoric and epistemic uncertainties.

Output Feedback Reachability of Controlled–Observable States for Nondeterministic Finite–State Systems

In this letter control design of nondeterministic finite state systems with reachability specifications is addressed. The class of controllers we use is rather general and combines feedforward and output feedback schemes. The proposed controller allows not only the state of the system to reach the desired target set but also the identification of which state of the target set has been reached. Necessary and sufficient conditions are derived for the control problem to admit a solution and a controller is designed. The solution to the investigated problem has important implications in the context of recovery control and symbolic control design of nonlinear and hybrid systems, as discussed also through some examples.

Deception in Supervisory Control

The use of deceptive strategies is important for an agent that attempts not to reveal his intentions in an adversarial environment. We consider a setting in which a supervisor provides a reference policy and expects an agent to follow the reference policy and perform a task. The agent may instead follow a different, deceptive policy to achieve a different task. We model the environment and the behavior of the agent with a Markov decision process, represent the tasks of the agent and the supervisor with reachability specifications, and study the synthesis of optimal deceptive policies for such agents. We also study the synthesis of optimal reference policies that prevent deceptive strategies of the agent and achieve the supervisor's task with high probability. We show that the synthesis of optimal deceptive policies has a convex optimization problem formulation, while the synthesis of optimal reference policies requires solving a nonconvex optimization problem. We also show that the synthesis of optimal reference policies is NP-hard.

Formal Reachability Analysis for Multi-Agent Reinforcement Learning Systems

Reachability analysis is one of the most basic and challenging problems in verification. We investigate this problem in multi-agent reinforcement learning (MARL) system by transforming the reachability analysis to decision-making problem tackled by mixed integer linear programming (MILP) solver Gurobi. We define syntax and semantics for multi-role strategy logic (mrSL) which is used to describe the reachability specification. The logic mrSL is a variant of SL to express properties, which provide a holistic perspective to describe reachability properties instead of specifying a specific agent reaches a certain state in the MARL system. And the algorithms to translate reachability property into MILP constraints are provided. A tool called MAReachAnalysis is introduced, which uses Gurobi to solve the corresponding reachability problem and evaluate it on the predator-prey task of multi-agent deep deterministic policy gradient (MADDPG) example, discussion of the experimental results obtained on a range of test cases.

Bisimulation Equivalence of Discrete-Time Stochastic Linear Control Systems

In this paper, we propose a definition of equivalence via stochastic bisimulation for the class of discrete-time stochastic linear control systems with possibly degenerate normally distributed disturbances. The notion is inspired by the notion of probabilistic bisimulation for probabilistic chains. Geometric necessary and sufficient conditions for checking this notion are derived. Model reduction via Kalman-like decomposition is also proposed. Connections with stochastic reachability are discussed and with finite horizon stochastic safety problems established. A discussion on the use of stochastic bisimulation equivalence for control design is given, and an application to optimal control problems with stochastic reachability specifications is finally presented.

The Linear Programming Approach to Reach-Avoid Problems for Markov Decision Processes

One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.

Minimum Resource Commitment for Reachability Specifications in a Discrete Time Linear Setting

This paper addresses control input design for a discrete time linear system. The goal is to satisfy a reachability specification and, at the same time, minimize the number of inputs that need to be set (influential inputs). To this purpose, we introduce an appropriate input parametrization so that, depending on the parameter values, some of the inputs act as control variables, while the others are treated as disturbances and can take an arbitrary value in their range. We then enforce the specification while maximizing the number of disturbance inputs. Two approaches are developed: one based on an open-loop scheme and one based on a compensation scheme. In the former, we end up solving a linear program. In the latter, the parametrization is extended so as to allow the influential inputs to depend on the non-influential ones, and the problem is reduced to a mixed integer linear program. A comparison between the two approaches is carried out, showing the superiority of the latter. Possible applications to system design and security of networked control systems are briefly discussed in the introduction.

Falsification of combined invariance and reachability specifications in hybrid control systems

We propose an abstraction-based method that can be applied to falsify a class of computation tree logic (CTL) specifications that combine invariance and reachability requirements in terms of the discrete state of a hybrid control system. The fragment of CTL that we address is not expressible in ACTLź (which includes LTL). The method involves applying supervisory control to a finite abstraction of the hybrid system to falsify the specification. For the class of systems that we consider, falsification of the specification implies a flaw in the design of the control and automation logic.

Using Computational Tree Logic Methods to Analyze Reachability in User Documentation

To support navigability, user documentation designers may use a table of contents, cross-References within procedures with page locations for related procedures, and in the case of PDF documentation, hyperlinks. Although standards and guidance exist for supporting navigable user documentation, problems appear across multiple industries including aviation and healthcare. To support user documentation development, we present a reachability analysis approach leveraging computational tree logic (CTL) reachability specifications as usability criteria coupled with a CTL model checker. A case study informed by an entry in an industry database highlights the effectiveness of the approach.

Distributed synthesis is simply undecidable

The distributed synthesis problem of safety and reachability languages is known to be undecidable. In this article, we establish that this is the case for very simple languages, namely for safety and reachability specifications in the intersection of LTL and ACTL.

Stochastic system controller synthesis for reachability specifications encoded by random sets

We consider a reach–avoid specification for a stochastic hybrid dynamical system defined as reaching a goal set at some finite time, while avoiding an unsafe set at all previous times. In contrast with earlier works which consider the target and avoid sets as deterministic, we consider these sets to be probabilistic. An optimal control policy is derived which maximizes the reach–avoid probability. Special structure on the stochastic sets is exploited to make the computation tractable for large space dimensions.

Low-complexity quantized switching controllers using approximate bisimulation

In this paper, we consider the problem of synthesizing low-complexity controllers for incrementally stable switched systems. For that purpose, we establish a new approximation result for the computation of symbolic models that are approximately bisimilar to a given switched system. The main advantage over existing results is that it allows us to design naturally quantized switching controllers for safety or reachability specifications; these can be pre-computed offline and therefore the online execution time is reduced. Then, we present a technique to reduce the memory needed to store the control law by borrowing ideas from algebraic decision diagrams for compact function representation and by exploiting the non-determinism of the synthesized controllers. We show the merits of our approach by applying it to a simple model of temperature regulation in a building.

Controller synthesis for safety and reachability via approximate bisimulation

In this paper, we consider the problem of controller design using approximately bisimilar abstractions with an emphasis on safety and reachability specifications. We propose abstraction-based approaches to controller synthesis for both types of specifications. We start by synthesizing a controller for an approximately bisimilar abstraction. Then, using a concretization procedure, we obtain a controller for our initial system that is proved “correct by design”. We provide guarantees of performance by giving estimates of the distance of the synthesized controller to the maximal (i.e., the most permissive) safety controller or to the time-optimal reachability controller. Finally, we use these techniques, combined with discrete approximately bisimilar abstractions of switched systems developed recently, for switching controller synthesis.

Safety and Reachability of Piecewise Linear Hybrid Dynamical Systems Based on Discrete Abstractions

In this paper, a novel methodology for analysis of piecewise linear hybrid systems based on discrete abstractions of the continuous dynamics is presented. An important characteristic of the approach is that the available control inputs are taken into consideration in order to simplify the continuous dynamics. Control specifications such as safety and reachability specifications are formulated in terms of partitions of the state space of the system. The approach provides a convenient general framework not only for analysis, but also for controller synthesis of hybrid systems. The research contributions of this paper impact the areas of analysis, verification, and synthesis of piecewise linear hybrid systems.

Controllers for reachability specifications for hybrid systems

The problem of systematically synthesizing hybrid controllers which satisfy multiple control objectives is considered. We present a technique, based on the principles of optimal control, for determining the class of least restrictive controllers that satisfies the most important objective (which we refer to as safety). The system performance with respect to lower priority objectives (which we refer to as efficiency) can then be optimized within this class. We motivate our approach by showing how the proposed synthesis technique simplifies to well-known results from supervisory control and pursuit evasion games when restricted to purely discrete and purely continuous systems respectively. We then illustrate the application of this technique to two examples, one hybrid (the steam boiler benchmark problem), and one primarily continuous (a flight vehicle management system with discrete flight modes).