Linear Temporal Logic Specifications Research Articles

On reward distribution in reinforcement learning of multi-agent surveillance systems with temporal logic specifications

In multi-agent systems, it is important to design a reward based on the contribution of each agent for efficient learning. In this paper, we propose a reward distribution method for a surveillance system based on our previously proposed multi-agent reinforcement learning method with an aggregator, in which a control specification is described by a linear temporal logic formula. In this method, the aggregator computes and distributes rewards according to the actions that agents take on the surveillance system. Finally, the effectiveness of the proposed method is presented through a numerical simulation of a surveillance problem addressing a specific type of linear temporal logic specification.

End-to-End Path Planning Under Linear Temporal Logic Specifications

End-to-End Path Planning Under Linear Temporal Logic Specifications

Zonotope-based Symbolic Controller Synthesis for Linear Temporal Logic Specifications

Zonotope-based Symbolic Controller Synthesis for Linear Temporal Logic Specifications

Certified reinforcement learning with logic guidance

Reinforcement Learning (RL) is a widely employed machine learning architecture that has been applied to a variety of control problems. However, applications in safety-critical domains require a systematic and formal approach to specifying requirements as tasks or goals. We propose a model-free RL algorithm that enables the use of Linear Temporal Logic (LTL) to formulate a goal for unknown continuous-state/action Markov Decision Processes (MDPs). The given LTL property is translated into a Limit-Deterministic Generalised Büchi Automaton (LDGBA), which is then used to shape a synchronous reward function on-the-fly. Under certain assumptions, the algorithm is guaranteed to synthesise a control policy whose traces satisfy the LTL specification with maximal probability.

A Specification-Guided Framework for Temporal Logic Control of Nonlinear Systems

This article proposes a specification-guided framework for control of nonlinear systems with linear temporal logic (LTL) specifications. In contrast with well-known abstraction-based methods, the proposed framework directly characterizes the winning set, i.e., the set of initial conditions from which a given LTL formula can be realized, over the continuous state space of the system via a monotonic operator. Following this characterization, an algorithm is proposed to practically approximate the operator via an adaptive interval subdivision scheme, which yields a finite-memory control strategy. We show that the proposed algorithm is sound for full LTL specifications, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">robustly complete</i> for specifications recognizable by deterministic Büchi automaton (DBA), the latter in the sense that control strategies can be found whenever the given specification can be satisfied with additional bounded disturbances. Without having to compute and store the abstraction and the resulting product system with the DBA, the proposed method is more memory efficient, which is demonstrated by complexity analysis and performance tests. A preprocessing stage is also devised to reduce computational cost via a decomposition of the specification. We show that the proposed method can effectively solve real-world control problems, such as jet engine compressor control and motion planning for manipulators and mobile robots.

Software Tool for Distribution of Linear Temporal Logic Specifications

This work presents the implementation of a software tool that ensures the execution of a global Linear Temporal Logic (LTL) specification for a team of mobile agents that are evolving in a static and known environment cluttered with regions of interest. The software is build on formalisms that extend our previous work, with the main contribution of handling strictly more expressive missions that also allow negations (or temporal avoidance) of some regions of interest. An algorithm is provided for decomposing the global LTL mission into independent robots’ tasks, together with a flow diagram of the entire method that solves the path planning problem. Simulations and comparative results are discussed for a case study, in order to support the usefulness of the software tool.

Optimal Probabilistic Motion Planning With Potential Infeasible LTL Constraints

This paper studies optimal motion planning subject to motion and environment uncertainties. By modeling the system as a probabilistic labeled Markov decision process (PL-MDP), the control objective is to synthesize a finite-memory policy, under which the agent satisfies complex high-level tasks expressed as linear temporal logic (LTL) with desired satisfaction probability. In particular, the cost optimization of the trajectory that satisfies infinite horizon tasks is considered, and the trade-off between reducing the expected mean cost and maximizing the probability of task satisfaction is analyzed. The LTL formulas are converted to limit-deterministic Büchi automata (LDBA) with a reachability acceptance condition and a compact graph structure. The novelty of this work lies in considering the cases where LTL specifications can be potentially infeasible and developing a relaxed product MDP between PL- MDP and LDBA. The relaxed product MDP allows the agent to revise its motion plan whenever the task is not fully feasible and quantify the revised plan’s violation measurement. A multi- objective optimization problem is then formulated to jointly consider the probability of task satisfaction, the violation with respect to original task constraints, and the implementation cost of the policy execution. The formulated problem can be solved via coupled linear programs. This work first bridges the gap between probabilistic planning revision of potential infeasible LTL specifications and optimal control synthesis of both plan prefix and plan suffix of the trajectory over the infinite horizons. Experimental results are provided to demonstrate the effectiveness of the proposed framework.

To Explore or Not to Explore: Regret-Based LTL Planning in Partially-Known Environments

In this paper, we investigate the optimal robot path planning problem for high-level specifications described by co-safe linear temporal logic (LTL) formulae. We consider the scenario where the map geometry of the workspace is partially-known. Specifically, we assume that there are some unknown regions, for which the robot does not know their successor regions a priori unless it reaches these regions physically. In contrast to the standard game-based approach that optimizes the worst-case cost, in the paper, we propose to use regret as a new metric for planning in such a partially-known environment. The regret of a plan under a fixed but unknown environment is the difference between the actual cost incurred and the best-response cost the robot could have achieved if it realizes the actual environment with hindsight. We provide an effective algorithm for finding an optimal plan that satisfies the LTL specification while minimizing its regret. A case study on firefighting robots is provided to illustrate the proposed framework. We argue that the new metric is more suitable for the scenario of partially-known environment since it captures the trade-off between the actual cost spent and the potential benefit one may obtain for exploring an unknown region.

Unified Multirate Control: From Low-Level Actuation to High-Level Planning

In this article, we present a hierarchical multirate control architecture for nonlinear autonomous systems operating in partially observable environments. Control objectives are expressed using syntactically co-safe linear temporal logic (LTL) specifications and the nonlinear system is subject to state and input constraints. At the highest level of abstraction, we model the system–environment interaction using a discrete mixed observable Markov decision process, where the environment states are partially observed. The high-level control policy is used to update the constraint sets and cost function of a model predictive controller (MPC) which plans a reference trajectory. Afterward, the MPC planned trajectory is fed to a low-level high-frequency tracking controller, which leverages control barrier functions to guarantee bounded tracking errors. Our strategy is based on model abstractions of increasing complexity and layers running at different frequencies. We show that the proposed hierarchical multirate control architecture maximizes the probability of satisfying the high-level specifications, while guaranteeing state and input constraint satisfaction. Finally, we tested the proposed strategy in simulations and experiments on examples inspired by the Mars exploration mission, where only partial environment observations are available.

Temporal Logic Task Allocation in Heterogeneous Multirobot Systems

We consider the problem of optimally allocating tasks, expressed as global linear temporal logic (LTL) specifications, to teams of heterogeneous mobile robots of different types. Each task may require robots of multiple types. To obtain a scalable solution, we propose a hierarchical approach that first allocates specific robots to tasks using the information about the tasks contained in the nondeterministic B <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ddot{\text{u}}$</tex-math></inline-formula> chi automaton (NBA) that captures the LTL specification and then designs low-level paths for robots that respect the high-level assignment. Specifically, motivated by “lazy collision checking” methods in robotics, we first prune and relax the NBA by removing all negative atomic propositions, which simplifies the planning problem by checking constraint satisfaction only when needed. Then, we extract sequences of subtasks from the relaxed NBA along with their temporal orders and formulate a mixed integer linear program to allocate these subtasks to robots. Finally, we define generalized multirobot path planning problems to obtain low-level paths that satisfy both the high-level task allocation and the constraints captured by the negative atomic propositions in the original NBA. We show that our method is complete for a subclass of LTL that covers a broad range of tasks and present numerical simulations demonstrating that it can generate paths with lower cost, considerably faster than existing methods.

Temporal Logic Planning and Receding Horizon Control for Signal Source Localization

This article copes with signal source localization by employing a receding horizon control approach with temporal logic planning in the light of a single mobile robot. First, a temporal logic planning approach is proposed such that the task requirements from the temporal logic specifications can be effectively dealt with based on the product automaton in an offline fashion. Second, in order to label the key nodes of the product automaton, a particle filter is utilized to predict the source positions as the key nodes. Third, on the basis of the product automaton, a receding horizon control approach with temporal logic planning is developed to produce the robot’s trajectory that satisfies a given linear temporal logic specification. Finally, the effectiveness of the proposed control approach is illustrated for signal source localization.

Temporal logic guided safe model-based reinforcement learning: A hybrid systems approach

This paper studies the problem of synthesizing control policies for uncertain continuous-time nonlinear systems from linear temporal logic (LTL) specifications using model-based reinforcement learning (MBRL). Rather than taking an abstraction-based approach, we view the interaction between the LTL formula’s corresponding Büchi automaton and the nonlinear system as a hybrid automaton whose discrete dynamics match exactly those of the Büchi automaton. To find satisfying control policies, we pose a sequence of optimal control problems associated with states in the accepting run of the automaton and leverage control barrier functions (CBFs) to prevent specification violation. Since solving many optimal control problems for a nonlinear system is computationally intractable, we take a learning-based approach in which the value function of each problem is learned online in real-time. Specifically, we propose a novel off-policy MBRL algorithm that allows one to simultaneously learn the uncertain dynamics of the system and the value function of each optimal control problem online while adhering to CBF-based safety constraints. Unlike related approaches, the MBRL method presented herein decouples convergence, stability, and safety, allowing each aspect to be studied independently, leading to stronger safety guarantees than those developed in related works. Numerical results are presented to validate the efficacy of the proposed method.

Little Tricky Logic: Misconceptions in the Understanding of LTL

Context: Linear Temporal Logic (LTL) has been used widely in verification. Its importance and popularity have only grown with the revival of temporal logic synthesis, and with new uses of LTL in robotics and planning activities. All these uses demand that the user have a clear understanding of what an LTL specification means. Inquiry: Despite the growing use of LTL, no studies have investigated the misconceptions users actually have in understanding LTL formulas. This paper addresses the gap with a first study of LTL misconceptions. Approach: We study researchers' and learners' understanding of LTL in four rounds (three written surveys, one talk-aloud) spread across a two-year timeframe. Concretely, we decompose "understanding LTL" into three questions. A person reading a spec needs to understand what it is saying, so we study the mapping from LTL to English. A person writing a spec needs to go in the other direction, so we study English to LTL. However, misconceptions could arise from two sources: a misunderstanding of LTL's syntax or of its underlying semantics. Therefore, we also study the relationship between formulas and specific traces. Knowledge: We find several misconceptions that have consequences for learners, tool builders, and designers of new property languages. These findings are already resulting in changes to the Alloy modeling language. We also find that the English to LTL direction was the most common source of errors; unfortunately, this is the critical "authoring" direction in which a subtle mistake can lead to a faulty system. We contribute study instruments that are useful for training learners (whether academic or industrial) who are getting acquainted with LTL, and we provide a code book to assist in the analysis of responses to similar-style questions. Grounding: Our findings are grounded in the responses to our survey rounds. Round 1 used Quizius to identify misconceptions among learners in a way that reduces the threat of expert blind spots. Rounds 2 and 3 confirm that both additional learners and researchers (who work in formal methods, robotics, and related fields) make similar errors. Round 4 adds deep support for our misconceptions via talk-aloud surveys. Importance This work provides useful answers to two critical but unexplored questions: in what ways is LTL tricky and what can be done about it? Our survey instruments can serve as a starting point for other studies.

Probably Approximately Correct Learning in Adversarial Environments With Temporal Logic Specifications

Reinforcement learning (RL) algorithms have been used to learn how to implement tasks in uncertain and partially unknown environments. In practice, environments are usually uncontrolled and may affect task performance in an adversarial way. In this article, we model the interaction between an RL agent and its potentially adversarial environment as a turn-based zero-sum stochastic game. The task requirements are represented both qualitatively as a subset of linear temporal logic (LTL) specifications, and quantitatively as a reward function. For each case in which the LTL specification is realizable and can be equivalently transformed into a deterministic Büchi automaton, we show that there always exists a memoryless almost-sure winning strategy that is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula> -optimal for the discounted-sum objective for any arbitrary positive <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula> . We propose a probably approximately correct (PAC) learning algorithm that learns such a strategy efficiently in an online manner with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> unknown reward functions and unknown transition distributions. To the best of our knowledge, this is the first result on PAC learning in stochastic games with independent quantitative and qualitative objectives.

Temporal Logic Trees for Model Checking and Control Synthesis of Uncertain Discrete-Time Systems

We propose algorithms for performing model checking and control synthesis for discrete-time uncertain systems under linear temporal logic (LTL) specifications. We construct temporal logic trees (TLTs) from LTL formulae via reachability analysis. In contrast to automaton-based methods, the construction of the TLT is abstraction-free for infinite systems; that is, we do not construct discrete abstractions of the infinite systems. Moreover, for a given transition system and an LTL formula, we prove that there exist both a universal TLT and an existential TLT via minimal and maximal reachability analysis, respectively. We show that the universal TLT is an underapproximation for the LTL formula and the existential TLT is an overapproximation. We provide sufficient conditions and necessary conditions to verify whether a transition system satisfies an LTL formula by using the TLT approximations. As a major contribution of this work, for a controlled transition system and an LTL formula, we prove that a controlled TLT can be constructed from the LTL formula via a control-dependent reachability analysis. Based on the controlled TLT, we design an online control synthesis algorithm, under which a set of feasible control inputs can be generated at each time step. We also prove that this algorithm is recursively feasible. We illustrate the proposed methods for both finite and infinite systems and highlight the generality and online scalability with two simulated examples.

A learning-based synthesis approach of reward asynchronous probabilistic games against the linear temporal logic winning condition.

The traditional synthesis problem is usually solved by constructing a system that fulfills given specifications. The system is constantly interacting with the environment and is opposed to the environment. The problem can be further regarded as solving a two-player game (the system and its environment). Meanwhile, stochastic games are often used to model reactive processes. With the development of the intelligent industry, these theories are extensively used in robot patrolling, intelligent logistics, and intelligent transportation. However, it is still challenging to find a practically feasible synthesis algorithm and generate the optimal system according to the existing research. Thus, it is desirable to design an incentive mechanism to motivate the system to fulfill given specifications. This work studies the learning-based approach for strategy synthesis of reward asynchronous probabilistic games against linear temporal logic (LTL) specifications in a probabilistic environment. An asynchronous reward mechanism is proposed to motivate players to gain maximized rewards by their positions and choose actions. Based on this mechanism, the techniques of the learning theory can be applied to transform the synthesis problem into the problem of computing the expected rewards. Then, it is proven that the reinforcement learning algorithm provides the optimal strategies that maximize the expected cumulative reward of the satisfaction of an LTL specification asymptotically. Finally, our techniques are implemented, and their effectiveness is illustrated by two case studies of robot patrolling and autonomous driving.

Temporal Logic Guided Meta Q-Learning of Multiple Tasks

Reinforcement learning (RL) based approaches have enabled robots to perform various tasks. However, most existing RL algorithms focus on learning a particular task, without considering generalization to new tasks. To address this issue, by combining meta learning and reinforcement learning, we develop a meta Q-learning of multi-task (MQMT) framework where the robot effectively learns a meta model from a diverse set of training tasks and then generalizes the learned model to a new set of tasks that have never been encountered during training using only a small amount of additional data. Particularly, the multiple tasks are specified by co-safe linear temporal logic specification. As a semantics-preserving rewriting operation, LTL progression is exploited to decompose training tasks into learnable sub-goals, which not only enables simultaneous learning of multiple tasks, but also facilitates reward design by converting non-Markovian reward process to Markovian ones. Reward shaping is further incorporated into the reward design to relax the sparse reward issue. The simulation and experiment results demonstrate the effectiveness of the MQMT framework.

LTL cross entropy optimisation for quadcopter task orchestration

ABSTRACT This paper presents a task orchestration framework for multi-agent systems utilising linear temporal logic (LTL) and cross entropy optimisation, a stochastic optimisation technique used for rare-event sampling. We define task orchestration as a combination of task decomposition, allocation and planning for a quadcopter or team of quadcopters given a high-level specification. Specifically, we consider tasks that are complex and consist of environment constraints, system constraints, or both, that must be satisfied. We first approach motion planning for the single agent case where transition systems for the environment allow tasks to be developed as linear temporal logic (LTL) specifications. Trajectories are then generated via motion primitives for a single quadcopter and optimised via cross entropy to ensure optimal satisfaction of a cost function. We extend this work to the multi-agent case where a team of homogeneous quadcopters are considered to satisfy an LTL specification. In order to provide faster computations and initial cost-agnostic sampling, we formulate the online version of multi-agent task allocation via cross entropy for tasks specified in LTL specifications. The results of this framework are verified in simulation and experimentally with a team of quadcopters.

Trylock, a case for temporal logic and eternity variables

• A specification and implementation of Trylock are given. • The implementation is not correct for branching time (CTL). • The implementation is correct for linear time (LTL). • It follows that LTL is to be preferred over CTL. • The correctness is proved by means of eternity variables. An example is given of a software algorithm that implements its specification in linear time temporal logic (LTL), but not in branching time temporal logic (CTL). In LTL, a prophecy of future behaviour is needed to prove the simulation. Eternity variables are used for this purpose. The final phase of the proof is a refinement mapping in which two threads exchange roles. The example is a software implementation of trylock in a variation of the fast mutual exclusion algorithm of Lamport (1987). It has been used fruitfully for the construction of software algorithms for high performance mutual exclusion.

Distributed Motion Coordination for Multirobot Systems Under LTL Specifications

This article investigates the online motion coordination problem for a group of mobile robots moving in a shared workspace, each of which is assigned a linear temporal logic specification. Based on the realistic assumptions that each robot is subject to both state and input constraints and can have only local view and local information, a fully distributed multirobot motion coordination strategy is proposed. For each robot, the motion coordination strategy consists of three layers. An offline layer precomputes the braking area for each region in the workspace, the controlled transition system, and a so-called potential function. An initialization layer outputs an initially safely satisfying trajectory. An online coordination layer resolves conflicts when one occurs. The online coordination layer is further decomposed into three steps. First, a conflict detection algorithm is implemented, which detects conflicts with neighboring robots. Whenever conflicts are detected, a rule is designed to assign dynamically a planning order to each pair of neighboring robots. Finally, a sampling-based algorithm is designed to generate local collision-free trajectories for the robot, which at the same time guarantees the feasibility of the specification. Safety is proven to be guaranteed for all robots at any time. The effectiveness and the computational tractability of the resulting solution is verified numerically by two case studies.