Safety-critical Research Articles

Monitoring with verified guarantees

AbstractRuntime monitoring is generally considered a light-weight alternative to formal verification. In safety-critical systems, however, the monitor itself is a critical component. For example, if the monitor is responsible for initiating emergency protocols, as proposed in a recent aviation standard, then the safety of the entire system critically depends on the correctness of the monitor. In this paper, we present a verification extension to the Lola monitoring language that extends the efficient specification of the monitor with Hoare-style annotations that guarantee the correctness of the monitor specification. We add two new operators, assume and assert, which specify assumptions of the monitor and expectations on its output, respectively. The validity of the annotations is established by an integrated SMT solver. We report on experience in applying the approach to specifications from the avionics domain, where the annotation with assumptions and assertions has lead to the discovery of safety-critical errors in specifications. The errors range from incorrect default values in offset computations to complex algorithmic errors that result in unexpected temporal patterns. We also report how verified specifications can be monitored efficiently at runtime.

Reliability Assessment of Safety-Critical Systems of Nuclear Power Plant using Ordinary Differential Equations and Reachability Graph

This study proposes a novel and integrated approach incorporating reachability graph (RG) and ordinary differential equations (ODE) to assess the reliability of safety–critical systems (SCS). The suggested framework is divided into nine steps. The system is first modeled using the Petri net (PN) to generate the ODE and RG. The ODE solution can be used to compute the marking probabilities of PN model. The marking probability values can be utilized to measure the state probabilities of RG. The resulting RG contains various states of the system. The failure states of RG are identified to assess the system's reliability. The suggested approach is implemented on the Shutdown System of Nuclear Power Plant, and is validated using Brown & Lipow model. The obtained accuracy of 99.53905% in the evaluation of reliability validates the technique. The state-space explosion is one of the major problems associated with conventional stochastic approaches like Markov chains, which use assumed transition probabilities to quantify reliability between different states. The proposed strategy is capable of avoiding the problem of state-space explosion.

FTSC: Fault-tolerant scheduling and control co-design for distributed real-time system

Ensuring control stability is essential for safety-critical systems. Task redundancy is one commonly employed technique to enhance control performance in applications afflicted by intermittent faults. While increasing the number of replicas can improve control performance, it also introduces challenges in terms of scheduling and mapping. Prior research primarily focuses on controller design to enhance control performance, with limited consideration given to fault tolerance and scheduling in a combined manner. In this paper, we propose a co-design scheme of Fault-Tolerant Scheduling and Control (FTSC), which aims to provide stable control performance along with feasible mapping and scheduling solutions. We present a comprehensive formulation that encompasses mapping, scheduling, and fault tolerance, optimizing control performance into a Mixed Integer Linear Programming (MILP) framework. The problem of task redundancy can either be addressed through a heuristic algorithm efficiently or be solved in a MILP-based approach with a better solution. A case study is conducted to demonstrate the effectiveness of our proposed scheme, highlighting its ability to reduce system control costs while meeting real-time, reliability, and schedulability constraints.

Robust Metric Hybrid Planning in Stochastic Nonlinear Domains Using Mathematical Optimization

The deployment of automated planning in safety critical systems has resulted in the need for the development of robust automated planners that can (i) accurately model complex systems under uncertainty, and (ii) provide formal guarantees on the model they act on. In this paper, we introduce a robust automated planner that can represent such stochastic systems with metric specifications and constrained continuous-time nonlinear dynamics over mixed (i.e., real and discrete valued) concurrent action spaces. The planner uses inverse transform sampling to model uncertainty, and has the capability of performing bi-objective optimization to first enforce the constraints of the problem as best as possible, and second optimize the metric of interest. Theoretically, we show that the planner terminates in finite time and provides formal guarantees on its solution. Experimentally, we demonstrate the capability of the planner to robustly control four complex physical systems under uncertainty.

A Learning-Based Method for Computing Control Barrier Functions of Nonlinear Systems With Control Constraints

Verifying the safety of states and designing a safety controller are very important for safety critical systems (for example, robotic and automotive systems). Since the reachable set of a state is hard to calculate online, it is difficult to determine whether the current state will enter the unsafe set in the future. Thus, the safe set of a system is usually obtained by calculating the forward invariant set offline. For control affine systems, the control barrier function provides a linear safety constraint on the control, which helps us obtain safety control online. However, the computation of control barrier functions is recognized as difficult, and existing methods make over-approximations or are only applicable to problems without control constraints. In this paper, we propose a learning-based method to compute the control barrier function for nonlinear control affine systems with control constraints. The method consists of a constraint satisfaction (CS) algorithm and a set expansion (SE) algorithm. The CS algorithm computes the control barrier function that satisfies the constraints, and the SE algorithm expands the volume of the safe set. The numerical results demonstrate that our algorithm outperforms existing methods and can improve existing control barrier functions.

Critical Observability of Labeled Time Petri Net Systems

A time Petri net is said to be critically observable at a given time instant if the markings consistent with any observation at the time instant are included either in the set of critical markings or non-critical markings. This work studies the verification problem of critical observability for timed discrete event systems modeled by bounded labeled time Petri nets. The proposed method is a two-fold process: a preliminary verification of critical observability for the underlying logic labeled Petri net and a further verification considering the time constraint associated with each transition. The first step is based on the concurrent composition of a reachability graph of the logic net. If the logic net is critically observable, then the time net is also critically observable at any given time instant. Otherwise, the second step is to design an algorithm to compute all pairs of transition-class sequences that violate critical observability at the given time instant, and then a set of linear programming problems is exploited to check critical observability for the corresponding timed system. <i>Note to Practitioners</i>&#x2014;Timed discrete event systems provide a theoretical model for safety-critical real applications such as air traffic management, smart grid, and industrial control, which are vulnerable to malicious attack and destruction at some particular time instants such that a system may be misled to dangerous states. Critical observability of a timed discrete event system is a property with which the predefined dangerous states can be determined and detected from the observation by an observer at a given time instant. This research aims to offer a systematic approach to check critical observability for timed discrete event systems modeled by bounded labeled time Petri nets, which can also present some new ideas and insights for practitioners in the field of safety-critical systems.

An Orchestration Method for Integrated Multi-Disciplinary Simulation in Digital Twin Applications

In recent years, the methodology of Model-Based System Engineering (MBSE) has become relevant to the design of complex products, especially when safety critical systems need to be addressed. It allows, in fact, the deployment of product development directly through some digital models, allowing an effective traceability of requirements, being allocated upon the system functions, components, and parts. This approach enhances the designer capabilities in controlling the product development, manufacturing and after-market services. However, the application of such a methodology requires overcoming several technological barriers, especially in terms of models integration. The interoperability and management of several models—developed within different software to cover multiple levels of detail across several technical disciplines—is still very difficult, despite the level of maturation achieved by Systems Engineering. This paper describes a possible approach to provide such a connection between tools to allow a complete multi-disciplinary and heterogeneous simulation to analyse complex systems, such as safety-critical ones, which are typical of aerospace applications. Such an application is within a defined industrial context, placing particular attention on the compatibility of the approach with the legacy processes and tools.

Goal-Oriented Modeling of an Urban Subway Control System Using KAOS

The extent to which a safety-critical system, such as an urban subway control system, accomplishes its goals is a fundamental metric of its success. Identifying and assessing these goals should thus be one of the primary tasks in safety-critical system development. The breakdown of these systems may result in the loss of human lives and assets. The failure of these systems is caused by insufficient, incomplete, ambiguous, or conflicting requirements. Non-functional requirements are also separated from requirement specifications. Goal-oriented requirements engineering methodologies, such as KAOS, are used to tackle these challenges by providing adequate, complete, unambiguous, and consistent requirements in terms of goals. As a result, the KAOS approach is utilized in this article to construct a goal-oriented model of an urban subway control system. Variability and obstacle concerns are also addressed in this study.

Multi-objective ω-Regular Reinforcement Learning

The expanding role of reinforcement learning (RL) in safety-critical system design has promoted ω-automata as a way to express learning requirements—often non-Markovian—with greater ease of expression and interpretation than scalar reward signals. However, real-world sequential decision making situations often involve multiple, potentially conflicting, objectives. Two dominant approaches to express relative preferences over multiple objectives are: (1) weighted preference , where the decision maker provides scalar weights for various objectives, and (2) lexicographic preference , where the decision maker provides an order over the objectives such that any amount of satisfaction of a higher-ordered objective is preferable to any amount of a lower-ordered one. In this article, we study and develop RL algorithms to compute optimal strategies in Markov decision processes against multiple ω-regular objectives under weighted and lexicographic preferences. We provide a translation from multiple ω-regular objectives to a scalar reward signal that is both faithful (maximising reward means maximising probability of achieving the objectives under the corresponding preference) and effective (RL quickly converges to optimal strategies). We have implemented the translations in a formal reinforcement learning tool, Mungojerrie , and we present an experimental evaluation of our technique on benchmark learning problems.

Mission risk control via joint optimization of sampling and abort decisions.

Information-driven mission abort is an effective way to control the failure risk of safety-critical systems during mission executions. We investigate the optimal sampling and mission abort decisions of partially observable safety-critical systems, where the underlying system health state can only be revealed by sampling. In contrast to previous studies, we employ partial health information to jointly determine: (a) whether to execute sampling, and (b) when to abort the mission in a dynamic manner, so as to minimize the expected total cost incurred by sampling, mission failure, and system malfunction. Dynamic sampling and mission abort policies are devised following the belief state, whose optimization model is cast into the framework of a partially observable Markov decision process. Some structural insights with regard to the value function, control limit selection, and optimality existence are presented. The performance of the proposed sampling and abort policy is tested by numerical experiments, which are proved to outperform other heuristic abort policies in mission loss control.

Out-of-Distribution Detection Is Not All You Need

The usage of deep neural networks in safety-critical systems is limited by our ability to guarantee their correct behavior. Runtime monitors are components aiming to identify unsafe predictions and discard them before they can lead to catastrophic consequences. Several recent works on runtime monitoring have focused on out-of-distribution (OOD) detection, i.e., identifying inputs that are different from the training data. In this work, we argue that OOD detection is not a well-suited framework to design efficient runtime monitors and that it is more relevant to evaluate monitors based on their ability to discard incorrect predictions. We call this setting out-of-model-scope detection and discuss the conceptual differences with OOD. We also conduct extensive experiments on popular datasets from the literature to show that studying monitors in the OOD setting can be misleading: 1. very good OOD results can give a false impression of safety, 2. comparison under the OOD setting does not allow identifying the best monitor to detect errors. Finally, we also show that removing erroneous training data samples helps to train better monitors.

A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control room

In safety-critical automatic systems, safety can be compromised if operators lack engagement. Effective detection of undesirable engagement states can inform the design of interventions for enhancing engagement. However, the existing engagement measurement methods suffer from several limitations which damage their effectiveness in the work environment. A novel engagement evaluation methodology, which adopts Artificial Intelligence (AI) technologies, has been proposed. It was developed using motorway control room operators as subjects. Openpose and Open Source Computer Vision Library (OpenCV) were used to estimate the body postures of operators, then a Support Vector Machine (SVM) was utilised to build the engagement evaluation model based on discrete states of operator engagement. The average accuracy of the evaluation results reached 0.89 and the weighted average precision, recall, and F1-score were all above 0.84. This study emphasises the importance of specific data labelling when measuring typical engagement states, forming the basis for potential control room improvements. Practitioner summary: This study demonstrates an automatic, real-time, objective, and relatively unobtrusive method for measuring dynamic operator engagement states. Computer vision technologies were used to estimate body posture, then machine learning (ML) was utilised to build the engagement evaluation model. The overall evaluation shows the effectiveness of this framework. Abbreviations: AI: Artificial Intelligence; OpenCV: Open Source Computer Vision Library; SVM: Support Vector Machine; UWES: Utrecht Work Engagement Scale; ISA Engagement Scale: Intellectual, Social, Affective Engagement Scale; DSSQ: Dundee Stress State Questionnaire; SSSQ: Short Stress State Questionnaire; EEG: electroencephalography; ECG: Electrocardiography; VMOE: Video-based Measurement for Operator Engagement; CMU: Carnegie Mellon University; CNN: Convolutional Neural Network; 2D: two dimensional; ML: Machine learning.

Safety Verification of Nonlinear Systems with Bayesian Neural Network Controllers

Bayesian neural networks (BNNs) retain NN structures with a probability distribution placed over their weights. With the introduced uncertainties and redundancies, BNNs are proper choices of robust controllers for safety-critical control systems. This paper considers the problem of verifying the safety of nonlinear closed-loop systems with BNN controllers over unbounded-time horizon. In essence, we compute a safe weight set such that as long as the BNN controller is always applied with weights sampled from the safe weight set, the controlled system is guaranteed to be safe. We propose a novel two-phase method for the safe weight set computation. First, we construct a reference safe control set that constraints the control inputs, through polynomial approximation to the BNN controller followed by polynomial-optimization-based barrier certificate generation. Then, the computation of safe weight set is reduced to a range inclusion problem of the BNN on the system domain w.r.t. the safe control set, which can be solved incrementally and the set of safe weights can be extracted. Compared with the existing method based on invariant learning and mixed-integer linear programming, we could compute safe weight sets with larger radii on a series of linear benchmarks. Moreover, experiments on a series of widely used nonlinear control tasks show that our method can synthesize large safe weight sets with probability measure as high as 95% even for a large-scale system of dimension 7.

Multispectral Invisible Coating: Laminated Visible-Thermal Physical Attack against Multispectral Object Detectors Using Transparent Low-E Films

Multispectral object detection plays a vital role in safety-critical vision systems that require an around-the-clock operation and encounter dynamic real-world situations(e.g., self-driving cars and autonomous surveillance systems). Despite its crucial competence in safety-related applications, its security against physical attacks is severely understudied. We investigate the vulnerability of multispectral detectors against physical attacks by proposing a new physical method: Multispectral Invisible Coating. Utilizing transparent Low-e films, we realize a laminated visible-thermal physical attack by attaching Low-e films over a visible attack printing. Moreover, we apply our physical method to manufacture a Multispectral Invisible Suit that hides persons from the multiple view angles of Multispectral detectors. To simulate our attack under various surveillance scenes, we constructed a large-scale multispectral pedestrian dataset which we will release in public. Extensive experiments show that our proposed method effectively attacks the state-of-the-art multispectral detector both in the digital space and the physical world.

Synchronous Deterministic Parallel Programming for Multi-Cores with ForeC

Embedded real-time systems are tightly integrated with their physical environment. Their correctness depends both on the outputs and timeliness of their computations. The increasing use of multi-core processors in such systems is pushing embedded programmers to be parallel programming experts. However, parallel programming is challenging because of the skills, experiences, and knowledge needed to avoid common parallel programming traps and pitfalls. This article proposes the ForeC synchronous multi-threaded programming language for the deterministic, parallel, and reactive programming of embedded multi-cores. The synchronous semantics of ForeC is designed to greatly simplify the understanding and debugging of parallel programs. ForeC ensures that ForeC programs can be compiled efficiently for parallel execution and be amenable to static timing analysis. ForeC’s main innovation is its shared variable semantics that provides thread isolation and deterministic thread communication. All ForeC programs are correct by construction and deadlock free because no non-deterministic constructs are needed. We have benchmarked our ForeC compiler with several medium-sized programs (e.g., a 2.274-line ForeC program with up to 26 threads and distributed on up to 10 cores, which was based on a 2.155-line non-multi-threaded C program). These benchmark programs show that ForeC can achieve better parallel performance than Esterel, a widely used imperative synchronous language for concurrent safety-critical systems, and is competitive in performance to OpenMP, a popular desktop solution for parallel programming (which implements classical multi-threading, hence is intrinsically non-deterministic). We also demonstrate that the worst-case execution time of ForeC programs can be estimated to a high degree of precision.

Safety-critical computer vision: an empirical survey of adversarial evasion attacks and defenses on computer vision systems

Considering the growing prominence of production-level AI and the threat of adversarial attacks that can poison a machine learning model against a certain label, evade classification, or reveal sensitive data about the model and training data to an attacker, adversaries pose fundamental problems to machine learning systems. Furthermore, much research has focused on the inverse relationship between robustness and accuracy, raising problems for real-time and safety-critical systems particularly since they are governed by legal constraints in which software changes must be explainable and every change must be thoroughly tested. While many defenses have been proposed, they are often computationally expensive and tend to reduce model accuracy. We have therefore conducted a large survey of attacks and defenses and present a simple and practical framework for analyzing any machine-learning system from a safety-critical perspective using adversarial noise to find the upper bound of the failure rate. Using this method, we conclude that all tested configurations of the ResNet architecture fail to meet any reasonable definition of ‘safety-critical’ when tested on even small-scale benchmark data. We examine state of the art defenses and attacks against computer vision systems with a focus on safety-critical applications in autonomous driving, industrial control, and healthcare. By testing a combination of attacks and defenses, their efficacy, and their run-time requirements, we provide substantial empirical evidence that modern neural networks consistently fail to meet established safety-critical standards by a wide margin.

COMPUTER-AIDED DESIGN OF FAULT-TOLERANT HARDWARE ARCHITECTURES FOR AUTONOMOUS DRIVING SYSTEMS

AbstractFault-tolerant hardware architectures for autonomous vehicles can be implemented through redundancy, diversity, separation, self-diagnosis, and reconfiguration. These approaches can be coupled with majority redundancy through M-out-of-N independent system architectures. The development of fault- tolerant systems is of central importance in the launch of autonomous driving systems from level 4. The increasing complexity of electrical and electronic systems is challenging for the design of safety-critical systems. This work aims to develop a method to manage this complexity in product development and to use it to compare different types of architectures. The basis is a system consisting of sensors and microcontrollers. The reliability of all possible MooN configurations of the system is calculated automatically by numerically solving the master equation of the corresponding Markov chain. Subsequently, a software-based fault tree analysis enables more detailed modeling of the component structure. The results show that four-line architectures can provide suitable results and that the development effort for 2-ECU systems is higher than for 1-ECU systems with respect to the ISO 26262 target values.

Improvement of Geant4 Neutron-HP package: Doppler broadening of the neutron elastic scattering kernel

Whether it is for shielding applications or for safety criticality studies, numerically solving the neutron transport equation with a good accuracy requires to precisely estimate the Doppler broadened elastic scattering kernel in the thermal and epithermal energy range of neutrons travelling in a free gas. In Geant4, low energy neutrons are transported using evaluated data libraries handled by the Neutron High-Precision (Neutron-HP) package. Version 11.00.p03 of the code features in particular the Doppler broadened elastic scattering kernel, provided by the so-called ‘Sampling of the Velocity of the Target’ (SVT) method. However this latter fails for resonant heavy nuclei such as 238U and can severely impact the solving of the Boltzmann equation in fissile media. To overcome this shortcoming, the Doppler Broadening Rejection Correction (DBRC) method has been implemented in Geant4 and successfully validated with the reference Monte Carlo neutron transport code Tripoli-4® (version 11). This development will be taken into account in the next release of the code. All the validations have been performed with an automated benchmark tool which has been designed to support the quality assurance of the Geant4 Neutron-HP package. This tool is currently available on an ad hoc Gitlab repository and will be included in Geant4.

Towards Abstraction-based Probabilistic Program Analysis

Probabilistic programs that can represent both probabilistic and non-deterministic choices are useful for creating reliability models of complex safety-critical systems that interact with humans or external systems. Such models are often quite complex, so their analysis can be hindered by state-space explosion. One common approach to deal with this problem is the application of abstraction techniques. We present improvements for an abstraction-refinement scheme for the analysis of probabilistic programs, aiming to improve the scalability of the scheme by adapting modern techniques from qualitative software model checking, and make the analysis result more reliable using better convergence checks. We implemented and evaluated the improvements in our Theta model checking framework.

Distributed and Localized Model-Predictive Control–Part I: Synthesis and Implementation

The increasing presence of large-scale distributed systems highlights the need for scalable control strategies where only local communication is required. Moreover, in safety-critical systems it is imperative that such control strategies handle constraints in the presence of disturbances. In response to this need, we present the Distributed and Localized Model Predictive Control (DLMPC) algorithm for large-scale linear systems. DLMPC is a distributed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">closed-loop</i> model predictive control (MPC) scheme wherein only local state and model information needs to be exchanged between subsystems for the computation and implementation of control actions. We use the System Level Synthesis (SLS) framework to reformulate the centralized MPC problem, and show that this allows us to naturally impose localized communication constraints between sub-controllers. The structure of the resulting problem can be exploited to develop an Alternating Direction Method of Multipliers (ADMM) based algorithm that allows for distributed and localized computation of closed-loop control policies. We demonstrate that computational complexity of the subproblems solved by each subsystem in DLMPC is independent of the size of the global system. DLMPC is the first MPC algorithm that allows for the scalable computation and implementation of distributed closed-loop control policies, and deals with additive disturbances. In our companion paper, we show that this approach enjoys recursive feasibility and asymptotic stability.