Safety-critical Cyber-physical Systems Research Articles

Statistical Verification using Surrogate Models and Conformal Inference and a Comparison with Risk-Aware Verification

Uncertainty in safety-critical cyber-physical systems can be modeled using a finite number of parameters or parameterized input signals. Given a system specification in Signal Temporal Logic (STL), we would like to verify that for all (infinite) values of the model parameters/input signals, the system satisfies its specification. Unfortunately, this problem is undecidable in general. Statistical model checking (SMC) offers a solution by providing guarantees on the correctness of CPS models by statistically reasoning on model simulations. We propose a new approach for statistical verification of CPS models for user-provided distribution on the model parameters. Our technique uses model simulations to learn surrogate models , and uses conformal inference to provide probabilistic guarantees on the satisfaction of a given STL property. Additionally, we can provide prediction intervals containing the quantitative satisfaction values of the given STL property for any user-specified confidence level. We compare this prediction interval with the interval we get using risk estimation procedures. We also propose a refinement procedure based on Gaussian Process (GP)-based surrogate models for obtaining fine-grained probabilistic guarantees over sub-regions in the parameter space. This in turn enables the CPS designer to choose assured validity domains in the parameter space for safety-critical applications. Finally, we demonstrate the efficacy of our technique on several CPS models.

Out-of-Distribution Detection in Dependent Data for Cyber-Physical Systems with Conformal Guarantees

Uncertainty in the predictions of learning-enabled components hinders their deployment in safety-critical cyber-physical systems (CPS). A shift from the training distribution of a learning-enabled component (LEC) is one source of uncertainty in the LEC’s predictions. Detection of this shift or out-of-distribution (OOD) detection on individual datapoints has therefore gained attention recently. But in many applications, inputs to CPS form a temporal sequence. Existing techniques for OOD detection in time-series data for CPS either do not exploit temporal relationships in the sequence or do not provide any guarantees on detection. We propose using deviation from the in-distribution temporal equivariance as the non-conformity measure in conformal anomaly detection framework for OOD detection in time-series data for CPS. Computing independent predictions from multiple conformal detectors based on the proposed measure and combining these predictions by Fisher’s method leads to the proposed detector CODiT with bounded false alarms. CODiT performs OOD detection on fixed-length windows of consecutive time-series datapoints by using Fisher value of the input window. We further propose performing OOD detection on real-time time-series traces of variable lengths with bounded false alarms. This can be done by using CODiT to compute Fisher values of the sliding windows in the input trace and combining these values by a merging function. Merging functions such as Harmonic Mean, Arithmetic Mean, Geometric Mean, and Bonferroni Method, etc. can be used to combine Fisher values of the sliding windows in the input trace, and the combined value can be used for OOD detection on the trace with bounded false alarm rate guarantees. We illustrate the efficacy of CODiT by achieving state-of-the-art results in two case studies for OOD detection on fixed-length windows. The first one is on an autonomous driving system with perception (or vision) LEC. The second case study is on a medical CPS for walking pattern or GAIT analysis where physiological (non-vision) data is collected with force-sensitive resistors attached to the subject’s body. For OOD detection on variable length traces, we consider the same case studies on the autonomous driving system and medical CPS for GAIT analysis. We report our results with four merging functions on the Fisher values computed by CODiT on the sliding windows of the input trace. We also compare the false alarm rate guarantees by these four merging functions in the autonomous driving system case study. Code, data, and trained models are available at https://github.com/kaustubhsridhar/time-series-OOD.

Securing Pacemakers using Runtime Monitors over Physiological Signals

Wearable and implantable medical devices (IMDs) are increasingly deployed to diagnose, monitor, and provide therapy for critical medical conditions. Such medical devices are safety-critical cyber-physical systems (CPSs). These systems support wireless features introducing potential security vulnerabilities. Although these devices undergo rigorous safety certification processes, runtime security attacks are inevitable. Based on published literature, IMDs such as pacemakers and insulin infusion systems can be remotely controlled to inject deadly electric shocks and excess insulin, posing a threat to a patient’s life. While prior works based on formal methods have been proposed to detect potential attack vectors using different forms of static analysis, these have limitations in preventing attacks at runtime. This paper discusses a formal framework for detecting cyber-physical attacks on a pacemaker by monitoring its security policies at runtime. We propose a wearable device that senses the Electrocardiogram (ECG) and Photoplethysmogram (PPG) of the body to detect attacks in a pacemaker. To facilitate the design of this device, we map the security policies of a pacemaker w.r.t ECG and PPG, paving the way for designing formal verification monitors for pacemakers for the first time using multiple physiological signals. The proposed monitoring framework allows the synthesis of parallel monitors from a given set of desired security policies, where all the monitors execute concurrently and generate an alarm to the user in the case of policy violation. Our implementation and the performance evaluation results demonstrate the technical feasibility of designing such a wearable device for attack detection in pacemakers. This device is separate from the pacemaker, ensuring no need for re-certification of pacemakers. Our approach is amenable to the application of security patches when new attack vectors are detected, making the approach ideal for runtime monitoring of medical CPSs.

Model predictive monitoring of dynamical systems for signal temporal logic specifications

Online monitoring aims to evaluate or to predict, at runtime, whether or not the behaviors of a system satisfy some desired specification. It plays a key role in safety-critical cyber–physical systems. In this work, we propose a new monitoring approach, called model predictive monitoring, for specifications described by Signal Temporal Logic (STL) formulae. Specifically, we assume that the observed state traces are generated by an underlying dynamical system whose model is known but the control law is unknown. The main idea is to use the dynamic of the system to predict future states when evaluating the satisfaction of the STL formulae. To this end, effective approaches for the computation of feasible sets of STL formulae are provided. We show that, by explicitly utilizing the model information of the dynamical system, the proposed online monitoring algorithm can falsify or certify of the specification in advance compared with existing algorithms, where no model information is used. We also demonstrate the proposed monitoring algorithm by several real world case studies.

Integrated Attack Tree in Residual Risk Management Framework

Safety-critical cyber-physical systems (CPSs), such as high-tech cars having cyber capabilities, are highly interconnected. Automotive manufacturers are concerned about cyber attacks on vehicles that can lead to catastrophic consequences. There is a need for a new risk management approach to address and investigate cybersecurity risks. Risk management in the automotive domain is challenging due to technological improvements and advances every year. The current standard for automotive security is ISO/SAE 21434, which discusses a framework that includes threats, associated risks, and risk treatment options such as risk reduction by applying appropriate defences. This paper presents a residual cybersecurity risk management framework aligned with the framework presented in ISO/SAE 21434. A methodology is proposed to develop an integrated attack tree that considers multiple sub-systems within the CPS. Integrating attack trees in this way will help the analyst to take a broad perspective of system security. Our previous approach utilises a flow graph to calculate the residual risk to a system before and after applying defences. This paper is an extension of our initial work. It defines the steps for applying the proposed framework and using adaptive cruise control (ACC) and adaptive light control (ALC) to illustrate the applicability of our work. This work is evaluated by comparing it with the requirements of the risk management framework discussed in the literature. Currently, our methodology satisfies more than 75% of their requirements.

Unified Graphical Co-modelling, Analysis and Verification of Cyber-physical Systems by Combining AADL and Simulink/Stateflow

The design of safety-critical cyber-physical systems (CPSs) involve several dimensions, including physics, hardware rchitecture and software functionality. It is desirable to design CPSs by taking these issues into account uniformly and yet, few existing design workflows support this aim. For instance, AADL is an architecturecentric modelling formalism for CPSs, which focuses on modelling architecture and prototyping real-time hardware platforms, but it delegates physical and software behavioral models to so-called annexes. By contrast, Simulink/Stateflow (S/S) focuses on modelling interacting physical and software behaviors, but does not render the non-functional characteristics of their hardware platforms. To address this issue, in [1], we proposed the combination of AADL and S/S, called AADL S/S, to comodel CPSs and presented a method to uniformly analyse and verify them. AADL S/S provides a unified graphical co-modelling environment for CPS design and supports simulation through C code generation. Also, [1] presented a formal semantics of AADL S/S by translation to Hybrid Communicating Sequential Processes (HCSP), yielding a deductive verification framework of the combined models using Hybrid Hoare Logic (HHL). Additionally, [1] proved the correctness of the translation of AADL S/S to HCSP.

Model Based Verification of Spiking Neural Networks in Cyber Physical Systems

Spiking Neural Networks (SNNs) have found increasing utility in designing safety-critical Cyber-Physical Systems (CPSs) such as implantable medical devices, autonomous vehicles, and space robotics due to their capability to operate on information represented in temporal coding and exhibit various behavioural modalities. Thus, there has been recent interest in formally verifying their timing behaviours and providing soundness guarantees of their diverse characteristics. However, beyond the simplistic Leaky Integrate and Fire (LIF) model, which only mimics 3 spiking behaviours, there is a lack of unifying methodology in literature to verify complex dynamics of biological neurons exhibiting 20 spiking behaviours as demonstrated by the pioneering work of Izhikevich. There is also a complete lack of formulation for the verification of SNN-based systems. This paper bridges these gaps by proposing a model-based approach for designing SNN-based controllers in CPS. We propose sound structural transformations translating any spiking neuron into networks of Timed Automata (TA), model the complex Izhikevich neural model and formally verify all 20 timing behaviours it exhibits for the first time. We then present two case studies that were modelled as SNNs using our approach: the PID controller, and the Car-Following controller, and subsequently attempt static model checking and statistical verification of their generated TA models for safety guarantees.

A contract-based semantics and refinement for hybrid Simulink block diagrams

Simulink is widely used for modelling, simulating, and analysing cyber–physical systems (CPS). However, for safety-critical CPS, Simulink is insufficient to ensure safety since it lacks an official formal semantics to support compositional reasoning and verification. In this paper, we present a contract-based semantic model for hybrid Simulink block diagrams, including both discrete-time and continuous-time blocks. In our semantic formalisation, the semantics of a block is defined as a contract, and we define five operations on contracts, which are sequential composition, parallel composition, feedback composition, variable renaming, and variable hiding. We then develop a refinement calculus for hybrid Simulink block diagrams. Finally, we use a water tank system and a vehicle speed control system as case studies to demonstrate our approach.

Towards correctness proof for hybrid Simulink block diagrams

Cyber–physical systems (CPS) are often modelled using Simulink to simulate plant and controller behaviour by block diagrams. However, since the incomplete coverage of simulation, design using Simulink alone is insufficient for ensuring the correctness of the modelled system, especially in safety-critical CPS. The correctness is that the Simulink block diagrams satisfy the formal specification of the system being modelled. In this work, we present a contract-based method, which supports compositional reasoning and refinement, for proving the correctness of hybrid Simulink block diagrams (i.e., the block diagrams mixed continuous–discrete blocks and multi-rate discrete-time block diagrams). To this end, we first define a formal semantics for hybrid Simulink block diagrams. We then use the contract as a specification language. Moreover, we present a satisfaction relation between the block diagram and the contract, and a refinement relation between the contracts. We prove that the refinement relation preserves the correctness: if the hybrid Simulink block diagram satisfies the composition contract and the composition contract refines the system specification, the hybrid block diagram is correct relative to the system specification. Furthermore, the effectiveness of our method is illustrated by a vehicle driving control system.

Addressing uncertainty in the safety assurance of machine-learning

There is increasing interest in the application of machine learning (ML) technologies to safety-critical cyber-physical systems, with the promise of increased levels of autonomy due to their potential for solving complex perception and planning tasks. However, demonstrating the safety of ML is seen as one of the most challenging hurdles to their widespread deployment for such applications. In this paper we explore the factors which make the safety assurance of ML such a challenging task. In particular we address the impact of uncertainty on the confidence in ML safety assurance arguments. We show how this uncertainty is related to complexity in the ML models as well as the inherent complexity of the tasks that they are designed to implement. Based on definitions of uncertainty as well as an exemplary assurance argument structure, we examine typical weaknesses in the argument and how these can be addressed. The analysis combines an understanding of causes of insufficiencies in ML models with a systematic analysis of the types of asserted context, asserted evidence and asserted inference within the assurance argument. This leads to a systematic identification of requirements on the assurance argument structure as well as supporting evidence. We conclude that a combination of qualitative arguments combined with quantitative evidence are required to build a robust argument for safety-related properties of ML functions that is continuously refined to reduce residual and emerging uncertainties in the arguments after the function has been deployed into the target environment.

Autonomous Emergency Braking of Electric Vehicles With High Robustness to Cyber-Physical Uncertainties for Enhanced Braking Stability

Modern autonomous emergency braking (AEB) system is a typical safety-critical cyber-physical system (CPS) synthesizing the vehicular communications, control, and proception technologies. However, the control performance of braking can be easily deteriorated by the road adhesion saturation in physical environment and the multi-hop communication network-induced delays in cyber systems. Motivated by this, a new multi-hop loop delay analysis method and its associated upper-bound expression is proposed to scrutinize the system uncertainties, within the scope of CPS. Following this endeavor, a hierarchical cyber-physical control scheme for the AEB system is proposed to mitigate the adverse effects of road adhesion saturation and multi-hop communication network-induced delays. At the upper layer, a μ-adaptive time-to-collision (TTC) planning strategy is adopted to generate the desired acceleration for collision risk avoidance considering the road adhesion saturation. At the lower layer, an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -based linear quadratic regulator (LQR) is designed for acceleration tracking with strong robustness to the uncertainties of cyber system. Hardware-In-Loop (HIL) experiments validate that the proposed method is superior in terms of braking accuracy and the robustness to the system uncertainties.

A Security-Enabled Safety Assurance Framework for IoT-Based Smart Homes

The exponential growth of the Internet of Things (IoT) has paved the way for safety-critical cyber-physical systems to enter our everyday activities. While such systems have changed the way of our life, they brought new challenges that can adversely affect our life and the environment. Safety and security are two such challenges that can hamper the widespread adoption of new IoT applications. Due to a large number of connected devices and their ability to control critical physical assets, intended attacks on them and/or unintended failure events such as mechanical failure of devices, communication failure, and unforeseen bad interactions between connected devices may cause an IoT-based system to enter into unsafe and dangerous physical states. By considering the importance of safety and security of IoT systems, in this article, we present a security-enabled safety monitoring framework for IoT-based systems. In the proposed framework, we utilize design-time system analysis to create an executable monitoring model that enables run-time safety assurance provision for a system via collecting and analyzing operational data and evidence to determine the safety status of the system and then taking appropriate actions and securely communicating the safety status and recommended actions to the system users to minimize the risk of the system entering into an unsafe state.

Highly Accurate Clock Synchronization With Drift Correction for the Controller Area Network

Modern vehicles, that have to be considered as safety-critical cyber-physical systems, require highly accurate <i>clock synchronization</i> (CS) among their distributed computing devices. Since Controller Area Network (CAN) is the predominant in-vehicle communication bus, it is highly relevant to support CS for CAN. This article proposes an original CS method for distributed in-vehicle networks based on CAN with both <i>offset</i> and <i>drift correction</i> . While offset correction is performed based on timestamps in periodic reference messages (RMs), our new method benefits from the re-synchronization mechanism of the CAN bit timing to apply highly accurate drift correction. Our algorithm does not make any modifications to the CAN protocol but requires the measurement of the phase error from the CAN controller. We derive analytical bounds for the expected clock differences and further validate the practicability of the proposed method by comprehensive experiments. As the main result, our method achieves a clock accuracy below <inline-formula><tex-math notation="LaTeX">$2\;\mu$</tex-math></inline-formula> s independent of important parameters such as the bit rate, RM period, bus utilization and time-varying clock drifts.

Adversarial Attacks and Defenses for Deep-Learning-Based Unmanned Aerial Vehicles

The introduction of deep learning (DL) technology can improve the performance of cyber–physical systems (CPSs) in many ways. However, this also brings new security issues. To tackle these challenges, this article explores the vulnerabilities of DL-based unmanned aerial vehicles (UAVs), which are typical CPSs. Although many research works have been reported previously on adversarial attacks of DL models, only few of them are concerned about safety-critical CPSs, especially regression models in such systems. In this article, we analyze the problem of adversarial attacks against DL-based UAVs and propose two adversarial attack methods against regression models in UAVs. The experiments demonstrate that the proposed nontargeted and targeted attack methods both can craft imperceptible adversarial images and pose a considerable threat to the navigation and control of UAVs. To address this problem, adversarial training and defensive distillation methods are further investigated and evaluated, increasing the robustness of DL models in UAVs. To our knowledge, this is the first study on adversarial attacks and defenses against DL-based UAVs, which calls for more attention to the security and safety of such safety-critical applications.

Recent Advancements in Automated Vehicle Certification: How the Experience from the Nuclear Sector Contributed to Making Them a Reality

The current paper discusses the most recent advancements in automated vehicle (AV) certification and how existing regulations/best practices from the nuclear field helped make AVs a reality. In particular, three main pillars differentiate the newly devised certification frameworks from previous automotive regulations: the introduction of a safety management system, the adoption of in-service monitoring and reporting data logging systems, and the use of virtual testing to demonstrate the safety of the AV; a set of pillars that are also found in the nuclear practice. The argumentation is informed by relevant literature and shared experiences from the nuclear power plant and transportation fields where there are established safety practices to manage safety-critical cyber-physical systems. Although the nuclear and transportation fields might seem unrelated, strong synergies were found, including risk management approaches and operational data collection procedures, which supported the timely drafting of the new regulation for AVs. Nonetheless, some open challenges remain due to peculiar aspects of AVs that will need to be addressed in the near future. Namely, practical methodologies for the residual-risk calculation for the various Operational Design Domains (ODDs), the design of scalable monitoring techniques, and the definition of detailed procedures for the virtual testing tool qualification.

Physical Adversarial Attack on a Robotic Arm

Collaborative Robots (cobots) are regarded as highly safety-critical cyber-physical systems (CPSs) owing to their close physical interactions with humans. In settings such as smart factories, they are frequently augmented with AI. For example, in order to move materials, cobots utilize object detectors based on deep learning models. Deep learning, however, has been demonstrated as vulnerable to adversarial attacks: a minor change (noise) to benign input can fool the underlying neural networks and lead to a different result. While existing works have explored such attacks in the context of picture/object classification, less attention has been given to attacking neural networks used for identifying object locations, and demonstrating that this can actually lead to a physical attack in a real CPS. In this paper, we propose a method to generate adversarial patches for the object detectors of CPSs, in order to miscalibrate them and cause potentially dangerous physical effects. In particular, we evaluate our method on an industrial robotic arm for card gripping, demonstrating that it can be misled into clipping the operator’s hand instead of the card. To our knowledge, this is the first work to attack object locations and lead to an incident on human users by an actual system.

Efficient Safe Control via Deep Reinforcement Learning and Supervisory Control – Case Study on Multi-Robot Warehouse Automation

Safe control has recently attracted much attention due to its applications in safety-critical cyber-physical systems. Supervisory control theory (SCT) is a formal control method that provides correct-by-construction safety certificates, but is computationally inefficient when the number of system components is large. On the other hand, deep reinforcement learning (DRL) provides a toolbox of efficient algorithms to compute control decisions even for very large state space, but does not always guarantee safety. In this paper, we propose to synergize SCT and DRL into a new efficient safe control approach. Specifically, we first employ DRL algorithms to efficiently compute sub-optimal solutions which may be unsafe; then we convert the obtained solutions into a standard supervisory control problem with an automaton (plant model) and a set of unsafe states (safety specification); finally we use SCT to synthesize a supervisor with a safety certificate. A case study of multi-robot warehouse logistic automation is conducted to demonstrate the efficiency of this proposed approach.

SysML-based compositional verification and safety analysis for safety-critical cyber-physical systems

Safety-critical cyber-physical systems (SC-CPS) have the characteristics of distributed, heterogeneous, strong coupling of computing resources and physical resources. With the increased acceptance of Model-Driven Development (MDD) in the safety-critical domain, the SysML language has been broadly used. Increasing complexity results in the formal verification of the SysML models of SC-CPS often faces the so-called state-explosion problem. Moreover, safety analysis is also an important step to ensure the quality of SC-CPS. Thus, this article proposes an integrated SysML modelling and verification approach to cover specification of nominal behaviour and safety. First, an extension of SysML is presented, in which the contract information (i.e. Assume and Guarantee) is extended for SysML block diagrams and a Safety Profile is proposed to describe safety-related concepts. Second, the transformation from SysML to the compositional verification tool OCRA is given. Third, the safety analysis is achieved by translating the Safety Profile model into FTA (Fault Tree Analysis). Finally, the prototype tools including SysML2OCRA and SafetyProfile2FTA are represented, and the effectiveness of the method proposed in this paper is verified through actual industrial cases.

From LTL to rLTL monitoring: improved monitorability through robust semantics

Runtime monitoring is commonly used to detect the violation of desired properties in safety critical cyber-physical systems by observing its executions. Bauer et al. introduced an influential framework for monitoring Linear Temporal Logic (LTL) properties based on a three-valued semantics for a finite execution: the formula is already satisfied by the given execution, it is already violated, or it is still undetermined, i.e., it can still be satisfied and violated by appropriate extensions of the given execution. However, a wide range of formulas are not monitorable under this approach, meaning that there are executions for which satisfaction and violation will always remain undetermined no matter how it is extended. In particular, Bauer et al. report that 44% of the formulas they consider in their experiments fall into this category. Recently, a robust semantics for LTL was introduced to capture different degrees by which a property can be violated. In this paper we introduce a robust semantics for finite strings and show its potential in monitoring: every formula considered by Bauer et al. is monitorable under our approach. Furthermore, we discuss which properties that come naturally in LTL monitoring—such as the realizability of all truth values—can be transferred to the robust setting. We show that LTL formulas with robust semantics can be monitored by deterministic automata, and provide tight bounds on the size of the constructed automaton. Lastly, we report on a prototype implementation and compare it to the LTL monitor of Bauer et al. on a sample of examples.

Compositional Thinking in Cyberphysical Systems Theory

Engineering safe and secure cyber-physical systems requires system engineers to develop and maintain a number of model views, both dynamic and static, which can be seen as algebras. We posit that verifying the composition of requirement, behavioral, and architectural models using category theory gives rise to a strictly compositional interpretation of cyber-physical systems theory, which can assist in the modeling and analysis of safety-critical cyber-physical systems.