Safety-critical Scenarios Research Articles

Safedrive dreamer: Navigating safety–critical scenarios in autonomous driving with world models

Achieving stable and reliable autonomous driving in complex traffic environments while ensuring safety under unpredictable conditions is a critical challenge in autonomous driving technology. To address this issue, this study proposes the Safedrive Dreamer navigation framework, which aims to reduce the reliance on trial-and-error learning in real-world scenarios, thereby mitigating the risks associated with dynamic driving conditions and enhancing vehicle foresight. This framework integrates the predictive capabilities of world models with the constrained Markov decision process (CMDP) and safety reinforcement learning to accurately anticipate future environmental changes. This ensures the reliability of autonomous driving routes, thereby improving both safety and efficiency. Furthermore, to reduce trial-and-error costs in real-world applications, this study employs PAC-Bayesian methods to derive generalization error bounds between simulations and reality, enabling a more effective transfer of knowledge and experience from simulations to real-world scenarios. Validation experiments in simulated and real environments showed that Safedrive Dreamer significantly outperformed existing autonomous driving solutions by 3.8% in key safety metrics, excelling in collision avoidance and risk reduction. This study provides new insights into the integration of world models into decision-making processes to enhance decision-making capabilities in safety–critical applications, thereby contributing significantly to the improvement of autonomous driving safety and reliability.

A novel route to secure and low-power accelerators for edge computing

A novel route to secure and low-power accelerators for edge computing Dr. Fabio Pavanello, Researcher at CNRS/CROMA Laboratory, guides us through a novel route to secure and low-power accelerators for edge computing, or in other words, the NEUROPULS approach. The current exponential increase of devices at the edge demands ever-growing performance to process vast amounts of locally generated data daily. Specialised hardware solutions are under investigation to achieve ultra-low-power computation while being lightweight, cost-effective, and commercially scalable. Additionally, such solutions shall also be highly secure, e.g., for data integrity and confidentiality, and, depending on the application, should provide extremely low latencies, e.g., in safety-critical scenarios (autonomous driving, healthcare, aviation, etc.).

SAR: Sharpness-Aware minimization for enhancing DNNs’ Robustness against bit-flip errors

As Deep Neural Networks (DNNs) are increasingly deployed in safety-critical scenarios, there is a growing need to address bit-flip errors occurring in hardware, such as memory. These errors can lead to changes in DNN weights, potentially degrading the performance of deployed models and causing catastrophic consequences. Existing methods improve DNNs’ fault tolerance or robustness by modifying network size, structure, or inference and training processes. Unfortunately, these methods often enhance robustness at the expense of clean accuracy and introduce additional overhead during inference. To address these issues, we propose Sharpness-Aware Minimization for enhancing DNNs’ Robustness against bit-flip errors (SAR), which aims to leverage the intrinsic robustness of DNNs. We begin with a comprehensive investigation of DNNs under bit-flip errors, yielding insightful observations regarding the intensity and occurrence of such errors. Based on these insights, we identify that Sharpness-Aware Minimization (SAM) has the potential to enhance DNN robustness. We further analyze this potential through the relationship between SAM formulation and our observations, building a robustness-enhancing framework based on SAM. Experimental validation across various models and datasets demonstrates that SAR can effectively improve DNN robustness against bit-flip errors without sacrificing clean accuracy or introducing additional inference costs, making it a “double-win” method compared to existing approaches.

위험 시나리오에 대한 트랜스포머 네트워크 기반 자율주행차량의 충돌 및 궤적 예측 알고리즘 개발

위험 시나리오에 대한 트랜스포머 네트워크 기반 자율주행차량의 충돌 및 궤적 예측 알고리즘 개발

Design of Multimodal In-Vehicle Notifications at Highway-Rail Grade Crossings: A Perception Study

Up to 67% of accidents at Highway-Rail Grade Crossings (HRGCs) are due to motorists not stopping in time. To reduce the likelihood of vehicle-train accidents, the current study investigated the design of five novel visual and auditory in-vehicle notifications. An “Inform” notification was tested for “display time.” Notifications, “Slow,” “Intersection,” and “Stop” were tested for “display time” and “speech length”; a notification, “Over Tracks” was tested for “speech length” and “presentation order.” Twenty-six participants viewed driving simulator recordings, and rated the notifications at active rail crossing scenarios using Likert scales. The results showed that shorter speech length was preferred with delayed notification displays, and longer speech length was preferred with early notification displays. However, complex scenarios might require longer speech even with delayed displays, depending on additional variables such as driver speed or visibility. Additionally, the tradeoff between notifications being too urgent and startling should also be considered in safety-critical scenarios.

A Novel Adversarial Example Detection Method Based on Frequency Domain Reconstruction for Image Sensors.

Convolutional neural networks (CNNs) have been extensively used in numerous remote sensing image detection tasks owing to their exceptional performance. Nevertheless, CNNs are often vulnerable to adversarial examples, limiting the uses in different safety-critical scenarios. Recently, how to efficiently detect adversarial examples and improve the robustness of CNNs has drawn considerable focus. The existing adversarial example detection methods require modifying CNNs, which not only affects the model performance but also greatly enhances training cost. With the purpose of solving these problems, this study proposes a detection algorithm for adversarial examples that does not need modification of the CNN models and can simultaneously retain the classification accuracy of normal examples. Specifically, we design a method to detect adversarial examples using frequency domain reconstruction. After converting the input adversarial examples into the frequency domain by Fourier transform, the adversarial disturbance from adversarial attacks can be eliminated by modifying the frequency of the example. The inverse Fourier transform is then used to maximize the recovery of the original example. Firstly, we train a CNN to reconstruct input examples. Then, we insert Fourier transform, convolution operation, and inverse Fourier transform into the features of the input examples to automatically filter out adversarial frequencies. We refer to our proposed method as FDR (frequency domain reconstruction), which removes adversarial interference by converting input samples into frequency and reconstructing them back into the spatial domain to restore the image. In addition, we also introduce gradient masking into the proposed FDR method to enhance the detection accuracy of the model for complex adversarial examples. We conduct extensive experiments on five mainstream adversarial attacks on three benchmark datasets, and the experimental results show that FDR can outperform state-of-the-art solutions in detecting adversarial examples. Additionally, FDR does not require any modifications to the detector and can be integrated with other adversarial example detection methods to be installed in sensing devices to ensure detection safety.

A Framework for Selecting and Assessing Wearable Sensors Deployed in Safety Critical Scenarios.

Wearable sensors for psychophysiological monitoring are becoming increasingly mainstream in safety critical contexts. They offer a novel solution to capturing sub-optimal states and can help identify when workers in safety critical environments are suffering from states such as fatigue and stress. However, sensors can differ widely in their application, design, usability, and measurement and there is a lack of guidance on what should be prioritized or considered when selecting a sensor. The paper aims to highlight which concepts are important when creating or selecting a device regarding the optimization of both measurement and usability. Additionally, the paper discusses how design choices can enhance both the usability and measurement capabilities of wearable sensors. The hopes are that this paper will provide researchers and practitioners in human factors and related fields with a framework to help guide them in building and selecting wearable sensors that are well suited for deployment in safety critical contexts.

A review of Explainable Artificial Intelligence in healthcare

Explainable Artificial Intelligence (XAI) encompasses the strategies and methodologies used in constructing AI systems that enable end-users to comprehend and interpret the outputs and predictions made by AI models. The increasing deployment of opaque AI applications in high-stakes fields, particularly healthcare, has amplified the need for clarity and explainability. This stems from the potential high-impact consequences of erroneous AI predictions in such critical sectors. The effective integration of AI models in healthcare hinges on the capacity of these models to be both explainable and interpretable. Gaining the trust of healthcare professionals necessitates AI applications to be transparent about their decision-making processes and underlying logic. Our paper conducts a systematic review of the various facets and challenges of XAI within the healthcare realm. It aims to dissect a range of XAI methodologies and their applications in healthcare, categorizing them into six distinct groups: feature-oriented methods, global methods, concept models, surrogate models, local pixel-based methods, and human-centric approaches. Specifically, this study focuses on the significance of XAI in addressing healthcare-related challenges, underscoring its vital role in safety-critical scenarios. Our objective is to provide an exhaustive exploration of XAI's applications in healthcare, alongside an analysis of relevant experimental outcomes, thereby fostering a holistic understanding of XAI's role and potential in this critical domain.

Probabilistic reach-avoid for Bayesian neural networks

Model-based reinforcement learning seeks to simultaneously learn the dynamics of an unknown stochastic environment and synthesise an optimal policy for acting in it. Ensuring the safety and robustness of sequential decisions made through a policy in such an environment is a key challenge for policies intended for safety-critical scenarios. In this work, we investigate two complementary problems: first, computing reach-avoid probabilities for iterative predictions made with dynamical models, with dynamics described by Bayesian neural network (BNN); second, synthesising control policies that are optimal with respect to a given reach-avoid specification (reaching a “target” state, while avoiding a set of “unsafe” states) and a learned BNN model. Our solution leverages interval propagation and backward recursion techniques to compute lower bounds for the probability that a policy's sequence of actions leads to satisfying the reach-avoid specification. Such computed lower bounds provide safety certification for the given policy and BNN model. We then introduce control synthesis algorithms to derive policies maximizing said lower bounds on the safety probability. We demonstrate the effectiveness of our method on a series of control benchmarks characterized by learned BNN dynamics models. On our most challenging benchmark, compared to purely data-driven policies the optimal synthesis algorithm is able to provide more than a four-fold increase in the number of certifiable states and more than a three-fold increase in the average guaranteed reach-avoid probability.

Safety evaluation and prediction of takeover performance in automated driving considering drivers’ cognitive load: A driving simulator study

Automated vehicles alleviate the need for driver attention and control. However, a takeover request (TOR) to the driver remains essential for emergencies and safety–critical scenarios beyond automation's capability. Thus, accessing drivers’ safety performance in various TOR scenarios is crucial for conditionally automated driving (SAE L3). However, TOR safety performance is seldom examined concerning drivers' cognitive load, despite its presumed relevance to TOR scenarios and human–machine interaction. Moreover, the adequacy of the time window preceding a TOR, critical for TOR safety prediction, remains inadequately explored. This study aims to assess safety performance across diverse TOR scenarios in Level 3 conditional automation, incorporating drivers’ cognitive load, and predict TOR safety by considering the time window's impact. A driving simulator experiment gathered eye movement and driving behavior data from 37 recruited participants. Participants were instructed to take control of the vehicle from automated driving within a time budget (TB) of 3 s or 7 s in obstacle avoidance (OA) or lane keeping (LK) scenarios while engaging in non-driving-related tasks (NDRTs). Participants’ subjective cognitive load in the different TOR scenarios was scaled using NASA-TLX. Furthermore, safe TOR performance was predicted utilizing Convolutional Neural Networks (CNNs) across different time window sizes preceding a TOR. The results indicate that: 1) cognitive loads in takeover scenarios ranked from highest to lowest as TB = 3s_OA, TB = 3s_LK, TB = 7s_OA, TB = 7s_LK; and the cognitive loads of the NDRTs ranked from highest to lowest as mistake finding, texting, chatting, monitoring; 2) the takeover safety performance in the four scenarios from lowest to highest was TB = 3s_OA, TB = 3s_LK, TB = 7s_OA, TB = 7s_LK; likewise, the takeover safety performance during the four NDRTs ranked from lowest to highest as mistake finding, monitoring, texting, chatting; 3) the time window size before the TORs significantly affected the prediction performance of the model. A 30-second window was recommended as optimal for predicting takeover safety using the CNN model, achieving an average F1 score of 0.8120 and 81.98 % accuracy. This study's findings enhance our comprehension of driving behavior characteristics during TOR and offer valuable insights for detecting driver states in conditionally automated driving contexts.

VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch

AI's widespread integration has led to neural networks (NN) deployment on edge and similar limited-resource platforms for safety-critical scenarios. Yet, NN's fragility raises concerns about reliable inference. Moreover, constrained platforms demand compact networks. This study introduces VeriCompress, a tool that automates the search and training of compressed models with robustness guarantees. These models are well-suited for safety-critical applications and adhere to predefined architecture and size limitations, making them deployable on resource-restricted platforms. The method trains models 2-3 times faster than the state-of-the-art approaches, surpassing them by average accuracy and robustness gains of 15.1 and 9.8 percentage points, respectively. When deployed on a resource-restricted generic platform, these models require 5-8 times less memory and 2-4 times less inference time than models used in verified robustness literature. Our comprehensive evaluation across various model architectures and datasets, including MNIST, CIFAR, SVHN, and a relevant pedestrian detection dataset, showcases VeriCompress's capacity to identify compressed verified robust models with reduced computation overhead compared to current standards. This underscores its potential as a valuable tool for end users, such as developers of safety-critical applications on edge or Internet of Things platforms, empowering them to create suitable models for safety-critical, resource-constrained platforms in their respective domains.

Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features

Ensuring the safety of autonomous vehicles (AVs) is of utmost importance, and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road’s structural features (lanes, turns, road speed, etc.). In this paper, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2 D . This visualisation helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritisation.

Real-time MPC with Control Barrier Functions for Autonomous Driving using Safety Enhanced Collocation

The autonomous driving industry is continuously dealing with safety-critical scenarios, and nonlinear model predictive control (NMPC) is a powerful control strategy for handling such situations. However, standard safety constraints are not scalable and require a long NMPC horizon. Moreover, the adoption of NMPC in the automotive industry is limited by the heavy computation of numerical optimization routines. To address those issues, this paper presents a real-time capable NMPC for automated driving in urban environments, using control barrier functions (CBFs). Furthermore, the designed NMPC is based on a novel collocation transcription approach, named RESAFE/COL, that allows to reduce the number of optimization variables while still guaranteeing the continuous time (nonlinear) inequality constraints satisfaction, through regional convex hull approximation. RESAFE/COL is proven to be 5 times faster than multiple shooting and more tractable for embedded hardware without a decrease in the performance, nor accuracy and safety of the numerical solution. We validate our NMPC-CBF with RESAFE/COL on digital twins of the vehicle and the urban environment and show the safe controller's ability to improve crash avoidance by 91%. Supplementary visual material can be found at https://youtu.be/_EnbfYwljp4.

Fear-Neuro-Inspired Reinforcement Learning for Safe Autonomous Driving.

Ensuring safety and achieving human-level driving performance remain challenges for autonomous vehicles, especially in safety-critical situations. As a key component of artificial intelligence, reinforcement learning is promising and has shown great potential in many complex tasks; however, its lack of safety guarantees limits its real-world applicability. Hence, further advancing reinforcement learning, especially from the safety perspective, is of great importance for autonomous driving. As revealed by cognitive neuroscientists, the amygdala of the brain can elicit defensive responses against threats or hazards, which is crucial for survival in and adaptation to risky environments. Drawing inspiration from this scientific discovery, we present a fear-neuro-inspired reinforcement learning framework to realize safe autonomous driving through modeling the amygdala functionality. This new technique facilitates an agent to learn defensive behaviors and achieve safe decision making with fewer safety violations. Through experimental tests, we show that the proposed approach enables the autonomous driving agent to attain state-of-the-art performance compared to the baseline agents and perform comparably to 30 certified human drivers, across various safety-critical scenarios. The results demonstrate the feasibility and effectiveness of our framework while also shedding light on the crucial role of simulating the amygdala function in the application of reinforcement learning to safety-critical autonomous driving domains.

Improving Risk-averse Distributional Reinforcement Learning with Adaptive Risk-seeking Exploration

In autonomous driving, reinforcement learning’s interactive learning capabilities have garnered interest. Among these, risk-averse distributional reinforcement learning has demonstrated promise in situations where safety is crucial. On the other hand, risk-averse policies degrade training performance by discouraging exploration. To provide an adaptive exploration policy, this work offers a novel framework for risk-averse distributional reinforcement learning. The proposed method is evaluated in a highly uncertain safety-critical scenario, motion planning for pedestrian avoidance in the face of uncertain interactions. The simulation results demonstrate improved safety, achieving training flexibility between exploration and exploitation.

“Braking bad”: The influence of haptic feedback and tram driver experience on emergency braking performance

Trams are experiencing a resurgence with worldwide network expansion driven by the need for sustainable and efficient cities. Trams often operate in shared or mixed-traffic environments, which raise safety concerns, particularly in hazardous situations. This paper adopts an international, mixed-methods approach, conducted through two interconnected studies in Melbourne (Australia) and Birmingham (UK). The first study involved qualitative interviews, while the second was an experimental study involving a virtual reality (VR) simulator and haptic master controller (i.e., speed lever). In tram operations, master controllers play a critical role in ensuring a smooth ride, which directly influences passenger safety and comfort. The objective was to understand how a master control system, enhanced with additional haptic feedback, could improve tram driver braking performance and perceptions in safety-critical scenarios. Interview results indicate that the use of the emergency brake is considered the final or ultimate choice by drivers, and their driving experience is a moderating factor in limiting its application. Combined with the experimental results, this paper highlights how implementing haptic feedback within a master controller can reduce the performance disparity between novice and experienced tram drivers.

Do we need high-order mutation in fault-based Boolean-specification testing?

Fault-based Boolean-specification testing is an essential weak mutation testing technique since the executing paths of the programs are usually dependent on those Boolean expressions in predicate statements. In mutation testing, the coupling effect hypothesis assumes that a test set that detects all the simple faults in the program can detect a high percentage of complex faults. Such a hypothesis suggests that we do not need to perform high-order mutation testing in practice. Many empirical results have shown the existence of the coupling effect. However, as fault-based Boolean-specification testing is mainly used in safety-critical scenarios, the question of whether high-order mutation testing is necessary for Boolean-specifications needs to be answered more strictly. To answer whether we need high-order mutation testing in fault-based Boolean-specification testing, we conduct an empirical study on general-form Boolean expressions. Experimental results show us that: (1) Generally, it is slightly easier to kill high-order mutants (HOMs) than first-order mutants (FOMs) since fault detection probabilities for complex faults (HOMs) are marginally higher than that for simple faults (FOMs) for almost all the cases. (2) However, for any fault types in Boolean-specifications, the coupling effect hypothesis is not guaranteed to hold with absolute certainty. (3) When HOMs involve the expression negation fault (ENF) or the operator reference fault (ORF), the percent of killed HOMs is lower. Such results suggest that, if the testing resource is sufficient, high-order mutation testing could be carried out on fault types ENF and ORF in fault-based Boolean-specification testing.

DUAL-C: Building a “soft error efficient” on-the-fly compression mechanism for raw video data at edge devices

Nowadays, various edge applications play an increasingly important role in our life, in which video data accounts for a dominant portion of the data traffic. Although standard compression methods, such as H.264 and MPEG4, can considerably reduce the wireless bandwidth for transmission energy savings of edge devices, the large volume of raw video streaming data is still buffered in edge device video buffer, constituting non-negligible energy consumption. Beyond energy issues, edge video data has been increasingly applied in safety–critical scenarios, where reliability has also become a critical concern. Unlike the controlled accuracy sacrifice of the lossy compression mechanism, the high information content per bit may render compressed data more sensitive to random transient faults attacking (i.e., soft errors), thereby inducing severe safety hazards in some video processing tasks (e.g., autonomous vehicle collision).Thus, in this paper, we propose a “soft error efficient” on-the-fly compression mechanism named DUAL-C to achieve energy savings in edge device video buffer while maintaining data reliability. Firstly, observing the duplicate feature of upper bits and approximable characteristic of lower bits within the camera sensors generated streaming raw pixel blocks, we design two compressing techniques: DU-C extracts the common duplicate upper-bits while AL-C merely stores one representative approximate lower-bits for all pixels within a block. By combining DU-C and AL-C with the Sampling-based Approximation paradigm (SA), DUAL-C achieves the overall energy consumption reduction of the video buffer by 71.31%. Secondly, to avoid the reliability “dark side” introduced by compression, we conduct a comprehensive error resilience analysis of compressed data and merely harden the error-critical control bits and common upper bits with ECC. Evaluation results show that DUAL-C compressed data exhibits even better error robustness than its uncompressed counterpart with negligible protection overhead.

The association between emotional intelligence and decision making for pilots

Emotional Intelligence (EI) refers to the regulation, perception, and management of self and others’ emotions. EI has been used to gain insights into decision making in corporate Human Resource Management (HRM) contexts, as well as in stressful situations. The link between EI and decision making in HRM could have great benefits to training and management in high-consequence and safety-critical industries. This research investigated the association between EI and decision making of pilots in the aviation industry. The aim was to uncover the level of association between EI dimensions and decision making for pilots; as well as to understand the role that pilots perceive EI dimensions play in their decision making in safety-critical scenarios. One hundred and seventeen pilots completed an online survey comprised of the Wong-Law EI Scale, decision making scenarios, and open-ended questions. The mixed-method analysis of the survey data showed a correlation between individual EI dimensions and decision-making scenarios, rather than the total scores. There are potential implications for general HRM research in EI and decision making as well as practical implications for the aviation industry. Overall, it was found that there is a link between EI and decision making, specifically for scenarios that involve other cognitive functions.

Guarantees for Real Robotic Systems: Unifying Formal Controller Synthesis and Reachset-Conformant Identification

Robots are used increasingly often in safety-critical scenarios, such as robotic surgery or human–robot interaction. To ensure stringent performance criteria, formal controller synthesis is a promising direction to guarantee that robots behave as desired. However, formally ensured properties only transfer to the real robot when the model is appropriate. In this article, we address this problem by combining the identification of a reachset-conformant model with controller synthesis. Since the reachset-conformant model contains all the measured behaviors of the real robot, the safety properties of the model transfer to the real robot. The transferability is demonstrated by experiments on a real robot, for which we synthesize tracking controllers.