Safety-critical Contexts Research Articles

Dynamic-Max-Value ReLU Functions for Adversarially Robust Machine Learning Models

The proliferation of deep learning has transformed artificial intelligence, demonstrating prowess in domains such as image recognition, natural language processing, and robotics. Nonetheless, deep learning models are susceptible to adversarial examples, well-crafted inputs that can induce erroneous predictions, particularly in safety-critical contexts. Researchers actively pursue countermeasures such as adversarial training and robust optimization to fortify model resilience. This vulnerability is notably accentuated by the ubiquitous utilization of ReLU functions in deep learning models. A previous study proposed an innovative solution to mitigate this vulnerability, presenting a capped ReLU function tailored to bolster neural network robustness against adversarial examples. However, the approach had a scalability problem. To address this limitation, a series of comprehensive experiments are undertaken across diverse datasets, and we introduce the dynamic-max-value ReLU function to address the scalability problem.

Neural networks in closed-loop systems: Verification using interval arithmetic and formal prover

Machine Learning approaches have been successfully used for the creation of high-performance control components of cyber–physical systems, where the control dynamics result from the combination of many subsystems. However, these approaches may lack the trustworthiness required to guarantee their reliable application in a safety-critical context. In this paper, we propose a combination of interval arithmetic and theorem-proving verification techniques to analyze safety properties in closed-loop systems that embed neural network components. We show the application of the proposed approach to a model-predictive controller for autonomous driving comparing the neural network verification performance with other existing tools. The results show that open-loop neural network verification through interval arithmetic can outperform existing approaches proving properties with a smaller time overhead. Furthermore, we demonstrate the capability of combining the two approaches to construct a formal model of the network in higher-order logic of the controlled system in a closed-loop.

A Framework for Selecting and Assessing Wearable Sensors Deployed in Safety Critical Scenarios.

Wearable sensors for psychophysiological monitoring are becoming increasingly mainstream in safety critical contexts. They offer a novel solution to capturing sub-optimal states and can help identify when workers in safety critical environments are suffering from states such as fatigue and stress. However, sensors can differ widely in their application, design, usability, and measurement and there is a lack of guidance on what should be prioritized or considered when selecting a sensor. The paper aims to highlight which concepts are important when creating or selecting a device regarding the optimization of both measurement and usability. Additionally, the paper discusses how design choices can enhance both the usability and measurement capabilities of wearable sensors. The hopes are that this paper will provide researchers and practitioners in human factors and related fields with a framework to help guide them in building and selecting wearable sensors that are well suited for deployment in safety critical contexts.

Evaluating Diffusion Models for the Automation of Ultrasonic Nondestructive Evaluation Data Analysis

We develop decision support and automation for the task of ultrasonic non-destructive evaluation data analysis. First, we develop a probabilistic model for the task and then implement the model as a series of neural networks based on Conditional Score-Based Diffusion and Denoising Diffusion Probabilistic Model architectures. We use the neural networks to generate estimates for peak amplitude response time of flight and perform a series of tests probing their behavior, capacity, and characteristics in terms of the probabilistic model. We train the neural networks on a series of datasets constructed from ultrasonic non-destructive evaluation data acquired during an inspection at a nuclear power generation facility. We modulate the partition classifying nominal and anomalous data in the dataset and observe that the probabilistic model predicts trends in neural network model performance, thereby demonstrating a principled basis for explainability. We improve on previous related work as our methods are self-supervised and require no data annotation or pre-processing, and we train on a per-dataset basis, meaning we do not rely on out-of-distribution generalization. The capacity of the probabilistic model to predict trends in neural network performance, as well as the quality of the estimates sampled from the neural networks, support the development of a technical justification for usage of the method in safety-critical contexts such as nuclear applications. The method may provide a basis or template for extension into similar non-destructive evaluation tasks in other industrial contexts.

Sparse subnetwork inference for neural network epistemic uncertainty estimation with improved Hessian approximation

Despite significant advances in deep neural networks across diverse domains, challenges persist in safety-critical contexts, including domain shift sensitivity and unreliable uncertainty estimation. To address these issues, this study investigates Bayesian learning for uncertainty handling in modern neural networks. However, the high-dimensional, non-convex nature of the posterior distribution poses practical limitations for epistemic uncertainty estimation. The Laplace approximation, as a cost-efficient Bayesian method, offers a practical solution by approximating the posterior as a multivariate normal distribution but faces computational bottlenecks in precise covariance matrix computation and storage. This research employs subnetwork inference, utilizing only a subset of the parameter space for Bayesian inference. In addition, a Kronecker-factored and low-rank representation is explored to reduce space complexity and computational costs. Several corrections are introduced to converge the approximated curvature to the exact Hessian matrix. Numerical results demonstrate the effectiveness and competitiveness of this method, whereas qualitative experiments highlight the impact of Hessian approximation granularity and parameter space utilization in Bayesian inference on mitigating overconfidence in predictions and obtaining high-quality uncertainty estimates.

Supporting Safety Analysis of Image-processing DNNs through Clustering-based Approaches

The adoption of deep neural networks (DNNs) in safety-critical contexts is often prevented by the lack of effective means to explain their results, especially when they are erroneous. In our previous work, we proposed a white-box approach (HUDD) and a black-box approach (SAFE) to automatically characterize DNN failures. They both identify clusters of similar images from a potentially large set of images leading to DNN failures. However, the analysis pipelines for HUDD and SAFE were instantiated in specific ways according to common practices, deferring the analysis of other pipelines to future work. In this paper, we report on an empirical evaluation of 99 different pipelines for root cause analysis of DNN failures. They combine transfer learning, autoencoders, heatmaps of neuron relevance, dimensionality reduction techniques, and different clustering algorithms. Our results show that the best pipeline combines transfer learning, DBSCAN, and UMAP. It leads to clusters almost exclusively capturing images of the same failure scenario, thus facilitating root cause analysis. Further, it generates distinct clusters for each root cause of failure, thus enabling engineers to detect all the unsafe scenarios. Interestingly, these results hold even for failure scenarios that are only observed in a small percentage of the failing images.

A review of physiological measures for mental workload assessment in aviation

AbstractThe relevant growth of human-machine interaction (HMI) systems in recent years is leading to the necessity of being constantly aware of the cognitive workload level of an operator, especially in a safety-critical context such as aviation. Since the confusion in the definition of this concept, this paper clarifies this terminology and also highlights its relationship with stress. Thus, we analysed the state-of-the-art of cognitive workload evaluations, showing three up-to-date methodologies: subjective, behavioural and physiological. In particular, the physiological approach is increasingly gaining attention in the literature due to today’s exponential growth of biomedical sensors. Therefore, a review of the most adopted physiological signals in the workload evaluation is provided, focusing on the aeronautical field. We conclude by highlighting the necessity of a multimodal approach for mental workload assessment as a result of this analysis.

Cooperation for Scalable Supervision of Autonomy in Mixed Traffic

Advances in autonomy offer the potential for dramatic positive outcomes in a number of domains, yet enabling their safe deployment remains an open problem. This work's motivating question is: In safety-critical settings, can we avoid the need to have one human supervise one machine at all times? The work formalizes this <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable supervision</i> problem by considering remotely located human supervisors and investigating how autonomous agents can cooperate to achieve safety. This article focuses on the safety-critical context of autonomous vehicles (AVs) merging into traffic consisting of a mixture of AVs and human drivers. The analysis establishes high reliability upper bounds on human supervision requirements. It further shows that AV cooperation can improve supervision reliability by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">orders of magnitude</i> and counterintuitively requires fewer supervisors (per AV) as more AVs are adopted. These analytical results leverage queuing-theoretic analysis, order statistics, and a conservative, reachability-based approach. A key takeaway is the potential value of cooperation in enabling the deployment of autonomy at scale. While this work focuses on AVs, the scalable supervision framework may be of independent interest to a broader array of autonomous control challenges.

Optimization of Landing Gear Mechanism using Generative Design

Abstract: Combining evolutionary algorithms with additive manufacturing (AM) techniques, new material approaches, in aerospace and industrial applications can result in advanced design procedures. This combination adds extra degrees of flexibility to the final design concept, enabling multifunctional designs. Aerospace, which is heavily focused on customised production, is a perfect fit for AM due to its effects on economies of scale and scope. These technologies are prepared to be included within the generative design process for safety-critical contexts, including the aerospace, thanks to novel structural materials and advanced AM processes. The three primary phases of conventional aircraft design are conceptual design, preliminary design, and detailed design. Multidisciplinary optimization processes are currently being developed to support designer in assessing the optimal solution. Generative Design is a novel form-finding process that takes into account structural performance, material properties and ergonomic demand. Evolutionary design approaches limits to numerical optimization, while Topology Optimization seeks to find an optimal structural configuration within a given design domain for specified objectives, constraints, loads and boundary conditions. This paper focuses on creating appropriate generative design models which can sustain same amount of stresses as of the original model. In the current study the optimization of Boeing 747’s nose landing gear is depicted using solidworks and fusion 360. Static structural analysis is utilized to evaluate the effects of stresses and failure mode produced by various materials, in order to figure out a materials characteristic as well as for material selection. Additionally, this paper examines original and generative design models

A teamwork effectiveness model for agile software development

Teamwork is crucial in software development, particularly in agile development teams which are cross-functional and where team members work intensively together to develop a cohesive software solution. Effective teamwork is not easy; prior studies indicate challenges with communication, learning, prioritization, and leadership. Nevertheless, there is much advice available for teams, from agile methods, practitioner literature, and general studies on teamwork to a growing body of empirical studies on teamwork in the specific context of agile software development. This article presents the agile teamwork effectiveness model (ATEM) for colocated agile development teams. The model is based on evidence from focus groups, case studies, and multi-vocal literature and is a revision of a general team effectiveness model. Our model of agile teamwork effectiveness is composed of shared leadership, team orientation, redundancy, adaptability, and peer feedback. Coordinating mechanisms are needed to facilitate these components. The coordinating mechanisms are shared mental models, communication, and mutual trust. We critically examine the model and discuss extensions for very small, multi-team, distributed, and safety-critical development contexts. The model is intended for researchers, team members, coaches, and leaders in the agile community.

MATTER: A tool for generating end-to-end IoT test scripts

In the last few years, Internet of Things (IoT) systems have drastically increased their relevance in many fundamental sectors. For this reason, assuring their quality is of paramount importance, especially in safety-critical contexts. Unfortunately, few quality assurance proposals for assuring the quality of these complex systems are present in the literature. In this paper, we extended and improved our previous approach for semi-automated model-based generation of executable test scripts. Our proposal is oriented to system-level acceptance testing of IoT systems. We have implemented a prototype tool taking in input a UML model of the system under test and some additional artefacts, and producing in output a test suite that checks if the system’s behaviour is compliant with such a model. We empirically evaluated our tool employing two IoT systems: a mobile health IoT system for diabetic patients and a smart park management system part of a smart city project. Both systems involve sensors or actuators, smartphones, and a remote cloud server. Results show that the test suites generated with our tool have been able to kill 91% of the overall 260 generated mutants (i.e. artificial bugged versions of the two considered systems). Moreover, the optimisation introduced in this novel version of our prototype, based on a minimisation post-processing step, allowed to reduce the time required for executing the entire test suites (about -20/25%) with no adverse effect on the bug-detection capability.

Uncertainty-Aware Prognosis via Deep Gaussian Process

The problem of Uncertainty Quantification (UQ) is of paramount importance when Machine Learning (ML) and Deep Learning (DL) models are deployed in the real world. In the context of safety-critical applications, such as the prediction of the Remaining Useful-Life (RUL) of infrastructure and industrial assets, the relevance of effective UQ approaches is even higher, given the potentially catastrophic consequences or substantial costs associated with maintenance decisions that are performed either too late or too early. Particularly in such safety critical application contexts, transparency and reliability are essential requirements and any ML-based solution not providing meaningful uncertainty estimates would not fully satisfy such desiderata. However, most ML and DL techniques used for RUL estimation are often not designed to perform UQ, thus limiting their effective applicability in real-world scenarios. To address this limitation, in this paper we investigate the performance of a recently proposed class of algorithms, Deep Gaussian Process (DGPs), for predicting the remaining useful lifetime (RUL). DGPs provide uncertainty estimates associated with their predictions, yet retaining the expressive power and learning capabilities of modern DL methods. DGPs are able to scale to very large datasets, which was a limitation of some of the previous algorithms but has increasingly become an essential requirement for many industrial applications and, in particular, for RUL prediction tasks for which UQ is of key importance. The main contribution of this paper is a thorough evaluation and comparison of several variants of DGPs applied to the problem of RUL prediction. The performance of the DGP algorithms is evaluated on the N-CMAPSS (New Commercial Modular Aero-Propulsion System Simulation) dataset from NASA for aircraft engines. The results demonstrate that DGPs are able to provide very accurate RUL predictions along with sensible uncertainty estimates, providing more reliable solutions for (safety-critical) real-life industrial applications.

Evaluating Impacts of Head Worn Displays on Teamwork in Emergency Response: Review of Challenges for the Field

Head Worn Displays (HWDs) are increasingly used to support mobile workers across diverse domains. However, little is known about how HWDs affect teamwork in safety-critical contexts. We conducted a narrative review examining the effects of HWDs on teamwork performance and team processes of situation awareness, communication, and coordination for emergency response. HWDs appear to improve the quality of team performance but increase time to perform under some conditions; effects on team processes are also mixed. It is difficult to compare results across studies due to the diverse technologies, contexts, and measures used. Successful design, evaluation, and deployment of HWDs in emergency response contexts may require a stronger foundation of theory-driven and process-based research. Perspectives such as joint cognitive systems, distributed cognition, and common ground may help researchers uncover mechanisms by which HWDs shape teamwork processes and how team processes affect team performance over time.

Assessing safety culture in a gas refinery complex: Development of a tool using a sociotechnical work systems and macroergonomics approach

Forty years of research into safety culture indicates that an appropriate assessment in a safety–critical context will include items that are common safety factors and also items that are specific to the particular industry and community. To develop a new survey tool, suitable for use in the gas refinery industry, a sociotechnical work systems approach was used, to capture all aspects of that work system context. The program was carried out in three parts at a gas refinery complex in the Persian Gulf. First, 18 semi-directed interviews of experienced employees were conducted. Using a confirmatory framework, the rich qualitative data were summarized into structural codes and 12 themes. Second, 15 safety specialists served as an expert panel on proposed items that emerged from the qualitative interviews. Their comments and assessment of content validity led to a 59-item questionnaire, in which the 12 themes were aligned to 12 safety culture factors and five macroergonomic subsystems of the sociotechnical work systems approach used. In the third part, the psychometric properties of the questionnaire were examined using a sample of 276 employees, supervisors and managers. The results of CFA indicated that a 56-item assessment tool was valid and reliable. Goodness-of-fit indices had acceptable values and measurement model was confirmed (CMIN/DF = 1.55; GFI = 0.94; AGFI = 0.91; CFI = 0.89; RMSEA = 0.05). The sociotechnical work system approach provided a suitable route to developing the safety assessment tool that recognizes aspects of the gas refinery industry, such as work schedule type, cultural environment, and being away from family support.

Examining the impact of ethical leadership on safety and task performance: a safety-critical context

PurposeAiming at understanding the effectiveness of leadership styles on workers' outcomes in safety-critical context, this study explores the impact of ethical leadership on safety and task performance under contingent effects of two safety-critical factors (i.e. perceived accident likelihood and perceived hazard exposure).Design/methodology/approachThe study is cross-sectional in nature and survey questionnaire was used for data collection. Data were collected from 397 workers from ten organizations producing chemical products. Multiple hierarchical regression was performed to test the hypothesized relationships.FindingsResults show that ethical leadership has positive association with workers' safety performance, safety attitude and task performance. Further, perceived accident likelihood moderated the influence of ethical leadership on workers' safety performance and attitude in such a way the association is strong when accident likelihood is high. Perceived hazard exposure moderated the link between ethical leadership and task performance such that higher hazard exposure decreases the effectiveness of ethical leadership.Practical implicationsFindings imply that managers can optimize employee safety for jobs associated with high safety-critical context through demonstration of ethical leadership behaviors. The study suggests that ethical leadership can prove to be important tool to improve workers' occupational safety well-being, which in turn helps them to improve their health and general well-being.Originality/valueContextualization of ethical leadership in safety-critical context is novelty of the study.

A systematic literature review of model-driven security engineering for cyber–physical systems

The last years have elevated the importance of cyber–physical systems like IoT applications, smart cars, or industrial control systems, and, therefore, these systems have also come into the focus of attackers. In contrast to software products running on PCs or smartphones, updating and maintaining cyber–physical systems presents a major challenge. This challenge, combined with the often decades-long lifetime of cyber–physical systems, and with their deployment in often safety-critical contexts, makes it particularly important to consider their security already at design time. When aiming to obtain a provably secure design, model-driven security approaches are key, as they allow to identify and mitigate threats in early phases of the development. As attacks may exploit both code-level as well as physical vulnerabilities, such approaches must consider not just the cyber layer but the physical layer as well. To find out which model-driven security approaches for cyber–physical systems exist considering both layers, we conducted a systematic literature review. From a set of 1160 initial papers, we extracted 69 relevant publications describing 17 candidate approaches. We found seven approaches specifically developed for cyber–physical systems. We provide a comprehensive description of these approaches, discuss them in particular detail, and determine their limitations. We found out that model-driven security is a relevant research area but most approaches focus only on specific security properties and even for CPS-specific approaches the platform is only rarely taken into account.

Should We Agree to Disagree? The Multilevel Moderated Relationship Between Safety Climate Strength And Individual Safety Motivation

Organisational research investigating climate perceptions often use constructs reflecting dispersion and disagreement, termed ‘climate strength’, to investigate situational pressures on behaviour expression. Within safety-specific contexts, research has tended to emphasise the prediction of climate strength rather than an examination of its effects on behaviour. The present paper investigates the important first pathway in the prediction of safety behaviour by investigating the influence of safety climate strength on the relationship between safety climate perceptions and individual safety motivation in a safety critical context using multilevel analyses. Contrary to expectations, results initially indicated that safety climate strength negatively influenced the relationship between safety climate perceptions and safety motivation, such that greater variability was associated with greater motivation. Post hoc analysis re-grouping responses into broader functional levels found support for an interaction, suggesting a difference in the scope of influence for safety climate strength between the two levels of analysis. These findings are discussed in light of self-determination theory, and suggestions for future research and practice made.

The great typography bake-off: comparing legibility at-a-glance

Typography plays an increasingly important role in today’s dynamic digital interfaces. Graphic designers and interface engineers have more typographic options than ever before. Sorting through this maze of design choices can be a daunting task. Here we present the results of an experiment comparing differences in glance-based legibility between eight popular sans-serif typefaces. The results show typography to be more than a matter of taste, especially in safety critical contexts such as in-vehicle interfaces. Our work provides both a method and rationale for using glanceable typefaces, as well as actionable information to guide design decisions for optimised usability in the fast-paced mobile world in which information is increasingly consumed in a few short glances. Practitioner summary: There is presently no accepted scientific method for comparing font legibility under time-pressure, in ‘glanceable’ interfaces such as automotive displays and smartphone notifications. A ‘bake-off’ method is demonstrated with eight popular sans-serif typefaces. The results produce actionable information to guide design decisions when information must be consumed at-a-glance. Abbreviations: DOT: department of transportation; FAA: Federal Aviation Administration; GHz: gigahertz; Hz: hertz; IEC: International Electrotechnical Commission; ISO: International Organization for Standardization; LCD: liquid crystal display; MIT: Massachusetts Institute of Technology; ms: milliseconds; OS: operating system

Decision Tree Learning for Uncertain Clinical Measurements

Clinical decision requires reasoning in the presence of imperfect data. DTs are a well-known decision support tool, owing to their interpretability, fundamental in safety-critical contexts such as medical diagnosis. However, learning DTs from uncertain data leads to poor generalization, and generating predictions for uncertain data hinders prediction accuracy. Several methods have suggested the potential of probabilistic decisions at the internal nodes in making DTs robust to uncertainty. Some approaches only employ probabilistic thresholds during evaluation. Others also consider the uncertainty in the learning phase, at the expense of increased computational complexity or reduced interpretability. The existing methods have not clarified the merit of a probabilistic approach in the distinct phases of DT learning, nor when the uncertainty is present in the training or the test data. We present a probabilistic DT approach that models measurement uncertainty as a noise distribution, independently realized: (1) when searching for the split thresholds, (2) when splitting the training instances, and (3) when generating predictions for unseen data. The soft training approaches (1, 2) achieved a regularizing effect, leading to significant reductions in DT size, while maintaining accuracy, for increased noise. Soft evaluation (3) showed no benefit in handling noise.

Factual and Counterfactual Explanations for Black Box Decision Making

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.