Safety-critical Areas Research Articles

European Nurses’ Burnout before and during the COVID-19 Pandemic and Its Impact on Patient Safety: A Scoping Review

Health systems around the world continue to experience healthcare workforce shortages, including shortages of nurses. This results in staff experiencing prolonged shifts and other stressors, which are linked to burnout, yet burned-out staff are then entrusted with the provision of patient care, despite healthcare facilities being regarded as safety-critical areas. It is assumed that the situation may have been worsened by the COVID-19 pandemic. This scoping review aims to identify the prevalence of burnout among nurses in Europe before and during the pandemic, the factors associated with this burnout, and its impact on patient safety. A literature search was conducted in the MEDLINE database; search terms included Nurse and Burnout and Patient Safety and their synonyms. The search limits used were English language, 2013 to 2023 publication years, original published research only (excluding review papers, dissertations, and unpublished reports), and studies conducted in European countries. A total of 16 papers were included for analysis: four indicated burnout levels of study participants, while 11 gave scores for individual burnout symptoms. Almost all studies indicated factors that were linked to burnout or individual dimensions of burnout. A few papers touched on the implications of these factors on patient safety. The review findings concluded that the few studies providing burnout levels on study participants could not give a clear picture of the burnout prevalence among nurses across Europe, either in general, before, or during the pandemic; therefore, this objective was not achieved and more research is required to establish this. Several factors thematically classified as sociodemographic, personal, organizational, and patient/client-related were linked to burnout. However, there were some contradictions in sociodemographic factors. Low personal accomplishment was the most common symptom of burnout before the pandemic, followed by emotional exhaustion and lastly depersonalization. Emotional exhaustion took the lead during the COVID-19 pandemic, followed by low personal accomplishment, and depersonalization remained the least common. More research is needed to establish the relationship between burnout and these factors.

Massively Parallel Evasion Attacks and the Pitfalls of Adversarial Retraining

Even with widespread adoption of automated anomaly detection in safety-critical areas, both classical and advanced machine learning models are susceptible to first-order evasion attacks that fool models at run-time (e.g. an automated firewall or an anti-virus application). Kernelized support vector machines (KSVMs) are an especially useful model because they combine a complex geometry with low run-time requirements (e.g. when compared to neural networks), acting as a run-time lower bound when compared to contemporary models (e.g. deep neural networks), to provide a cost-efficient way to measure model and attack run-time costs. To properly measure and combat adversaries, we propose a massively parallel projected gradient descent (PGD) evasion attack framework. Through theoretical examinations and experiments carried out using linearly-separable Gaussian normal data, we present (i) a massively parallel naive attack, we show that adversarial retraining is unlikely to be an effective means to combat an attacker even on linearly separable datasets, (ii) a cost effective way of evaluating models defences and attacks, and an extensible code base for doing so, (iii) an inverse relationship between adversarial robustness and benign accuracy, (iv) the lack of a general relationship between attack time and efficacy, and (v) that adversarial retraining increases compute time exponentially while failing to reliably prevent highly-confident false classifications.

Minimally Distorted Adversarial Images with a Step-Adaptive Iterative Fast Gradient Sign Method

The safety and robustness of convolutional neural networks (CNNs) have raised increasing concerns, especially in safety-critical areas, such as medical applications. Although CNNs are efficient in image classification, their predictions are often sensitive to minor, for human observers, invisible modifications of the image. Thus, a modified, corrupted image can be visually equal to the legitimate image for humans but fool the CNN and make a wrong prediction. Such modified images are called adversarial images throughout this paper. A popular method to generate adversarial images is backpropagating the loss gradient to modify the input image. Usually, only the direction of the gradient and a given step size were used to determine the perturbations (FGSM, fast gradient sign method), or the FGSM is applied multiple times to craft stronger perturbations that change the model classification (i-FGSM). On the contrary, if the step size is too large, the minimum perturbation of the image may be missed during the gradient search. To seek exact and minimal input images for a classification change, in this paper, we suggest starting the FGSM with a small step size and adapting the step size with iterations. A few decay algorithms were taken from the literature for comparison with a novel approach based on an index tracking the loss status. In total, three tracking functions were applied for comparison. The experiments show our loss adaptive decay algorithms could find adversaries with more than a 90% success rate while generating fewer perturbations to fool the CNNs.

Reward Certification for Policy Smoothed Reinforcement Learning

Reinforcement Learning (RL) has achieved remarkable success in safety-critical areas, but it can be weakened by adversarial attacks. Recent studies have introduced ``smoothed policies" to enhance its robustness. Yet, it is still challenging to establish a provable guarantee to certify the bound of its total reward. Prior methods relied primarily on computing bounds using Lipschitz continuity or calculating the probability of cumulative reward being above specific thresholds. However, these techniques are only suited for continuous perturbations on the RL agent's observations and are restricted to perturbations bounded by the l2-norm. To address these limitations, this paper proposes a general black-box certification method, called ReCePS, which is capable of directly certifying the cumulative reward of the smoothed policy under various lp-norm bounded perturbations. Furthermore, we extend our methodology to certify perturbations on action spaces. Our approach leverages f-divergence to measure the distinction between the original distribution and the perturbed distribution, subsequently determining the certification bound by solving a convex optimisation problem. We provide a comprehensive theoretical analysis and run experiments in multiple environments. Our results show that our method not only improves the tightness of certified lower bound of the mean cumulative reward but also demonstrates better efficiency than state-of-the-art methods.

Determination of quality classes for material extrusion additive manufacturing using image processing

Tools for implementing a systematic quality management are necessary for the use of material extrusion as an additive manufacturing process for products with high quality requirements. Well-defined quality classes are crucial for ensuring that the requirements for a product can be communicated transparently and that the existing properties can be evaluated. Furthermore, there is a lack of capable measurement equipment for the acquisition of process data during the production process. To address these challenges, the present paper introduces an image processing system that determines quality indicators for individual layers in terms of imperfect surface percentages and the number of imperfections. The central element of the hardware is an adaptive darkfield illumination, which leads to high-contrast images. In addition, five types of layer subareas are identified in a segmentation step. Unsupervised machine learning methods are then used to detect imperfections in each layer subarea. In the segmentation, the current layer can be distinguished from irrelevant image background regions with an F-measure of 0.981. For the layer-wise measurement of the quality indicators, relative measurement errors with standard deviations of 25 to 76.1% are found. After evaluating the capabilities of the image processing system, a proposal for limits of quality classes is derived by monitoring several material extrusion processes. For this purpose, three quality classes for each of the five layer subareas are deduced from the process scatter measured by the image processing system. The results are an important contribution to the industrialization of material extrusion in safety–critical areas such as medical technology or the aerospace industry.

LTCC-based Ceramic Substrates for Identification of Trustworthy Electronics

The technology of low temperature cofired ceramics (LTCC) is already established in the production of reliable and robust electronic assemblies (e.g., multi-chip modules, system-in-package) as well as in safety-critical areas. With the incorporation of additional features based on surface profiles modular protection functions and barriers will be developed as a contribution to trustworthy electronics. Both obvious and hidden structures are investigated in a modular way, which in combination ensures a one-to-one tamper-proof identification.

“I do not know! but why?” — Local model-agnostic example-based explanations of reject

Machine learning based decision making systems in safety critical areas place high demands on the accuracy and generalization ability of the underlying model. A common strategy to deal with uncertainties and possible mistakes is offered by learning with reject option, i.e. a model can refrain from prediction in ambiguous cases and leave the decision to a human expert. Yet, as for the models themselves, human decision-making is hampered by the fact that reject options are often implemented as black-box rules: Experts cannot readily understand the reasons for rejection.In this work, we propose a model-agnostic framework that enriches classification with reject option by explanation mechanisms. More specifically, we combine conformal prediction as a popular mathematically based technology of certainty estimation with local surrogates derived for the region of interest. This allows us to provide local explanations in terms of example-based explanation methods, including counterfactual, semi-factual, and factual methods. We demonstrate the performance of this technology through a series of benchmarks using 6 different data sets; the associated code is open source.11This article represents a substantially extended version of the conference paper Artelt et al. (2022). In addition to a more detailed explanation and derivation of the algorithms, we explore semi-factual and factual explanation methods and provide evaluations on realistic benchmark data sets.

Criticality of Prognostics in the Operations of Autonomous Aircraft

<div>This article addresses the design, testing, and evaluation of rigorous and verifiable prognostic and health management (PHM) functions applied to autonomous aircraft systems. These PHM functions—many deployed as algorithms—are integrated into a holistic framework for integrity management of aircraft components and systems that are subject to both operational degradation and incipient failure modes. The designer of a comprehensive and verifiable prognostics system is faced with significant challenges. Data (both baseline and faulted) that are correlated, time stamped, and appropriately sampled are not always readily available. Quantifying uncertainty, and its propagation and management, which are inherent in prognosis, can be difficult. High-fidelity modeling of critical components/systems can consume precious resources. Data mining tools for feature extraction and selection are not easy to develop and maintain. And finally, diagnostic and prognostic algorithms that address accurately the designer’s specifications are not easy to develop, verify, deploy, and sustain. These are just the technical challenges. On top of these are business challenges, for example, demonstrating that the PHM functionality will be economically beneficial to the system stakeholders, and finally, there are regulatory challenges, such as, assuring the authorities that the PHM system will have the necessary safety assurance levels while delivering its performance goals. This article tackles all three aspects of the use of PHM systems in autonomous systems. It outlines how some of the technical challenges have been overcome and demonstrates why PHM could be essential in this ecosystem and why regulatory authorities are increasingly open to the use of PHM systems even in the most safety-critical areas of aviation.</div>

RTA-IR: A runtime assurance framework for behavior planning based on imitation learning and responsibility-sensitive safety model

Current research on artificial intelligence (AI) algorithms in safety–critical areas remains extremely challenging due to their inability to be fully verified at design time. In this paper, we propose an RTA-IR architecture, which bypasses the formal verification of the AI algorithm by incorporating runtime assurance (RTA) and provides safety assurances for the AI controllers of complex autonomous vehicles (such as those obtained using neural networks) without excessive performance sacrifice. RTA-IR consists of a high-performance and unproven advanced controller and two verifiable safety controllers and a decision module designed based on the Responsibility Sensitive Safety Model (RSS). The advanced controller is designed based on attention generating adversarial imitation learning(GAIL), which can imitate the efficient policies of experts from a set of expert demonstrations. RSS provides verifiable safety criteria and switching logic for the decision module, and RTA-IR provides safety for autonomous vehicles when the advanced controller produces unsafe control, as well as restoring control of the vehicle by the advanced controller under conditions that confirm safety. We tested and evaluated RTA-IR separately for two levels of traffic density in one driving task. Experiments have shown that RTA-IR exhibits superior performance in terms of both safety and efficiency compared to the baseline method.

Verification of Safety for Synchronous-Reactive System Using Bounded Model Checking

Real-time embedded systems are increasingly applied in safety-critical areas, so guaranteeing the correctness of such systems by means of formal methods becomes particularly important. In this paper, we propose an optimized bounded model checking (BMC)-based formal verification approach for the verification of safety for synchronous-reactive (SR) models, which are often used to design systems with complicated control logic, especially the real-time embedded control systems. This method is based on the tackling of a series of challenging problems including the management of the logical clock, encoding of the contained ports, representation of the data types of ports, descriptions of behaviors of various components in a considered model, and formal consideration of the fixed-point semantics. We have implemented this proposed method in the prototype Ptolemy-Z3, and integrated this tool into the Ptolemy II environment. In addition, the experimental evaluation on 22 SR models has shown that our method performs better than the existing automatic verification method in Ptolemy II.

Saca-FI: A microarchitecture-level fault injection framework for reliability analysis of systolic array based CNN accelerator

As convolutional neural network CNN accelerators are being adopted in emerging safety-critical areas, their reliability becomes prominent. The systolic array is widely used as the major processing structure in CNN accelerators, so its reliability features need to be comprehensively explored. In this work, we develop a microarchitecture-level fault injection framework, saca-FI, to analyze the reliability of systolic array based CNN accelerators. It is able to investigate various vulnerability sources (e.g., single-bit flips, multi-bit flips, stuck-at faults, multiple faults) and at different levels (e.g., model layer and storage locations). Based on saca-FI, we evaluate the resilience of several commonly used networks, and observe several important architecture-level reliability characteristics. We then propose two opportunistic protections to demonstrate the usage of saca-FI in designing energy-efficient error protection mechanisms.

Explanation before Adoption: Supporting Informed Consent for Complex Machine Learning and IoT Health Platforms

Explaining health technology platforms to non-technical members of the public is an important part of the process of informed consent. Complex technology platforms that deal with safety-critical areas are particularly challenging, often operating within private domains (e.g. health services within the home) and used by individuals with various understandings of hardware, software, and algorithmic design. Through two studies, the first an interview and the second an observational study, we questioned how experts (e.g. those who designed, built, and installed a technology platform) supported provision of informed consent by participants. We identify a wide range of tools, techniques, and adaptations used by experts to explain the complex SPHERE sensor-based home health platform, provide implications for the design of tools to aid explanations, suggest opportunities for interactive explanations, present the range of information needed, and indicate future research possibilities in communicating technology platforms.

Trajectory evaluation using repeated rail-bound measurements

Abstract Many applications today require the precise determination of the position and orientation of a moving platform over time. However, especially in safety-critical areas, it is also important to derive quality characteristics of the trajectory estimation. This allows verification that sensors are operating within the precision and accuracy required for the application. In this paper, we propose a methodology for trajectory evaluation and address the challenges involved. Our approach is based on repeated measurements obtained using a closed loop rail track and allows the evaluation of the trajectory estimation in terms of precision and accuracy. Starting with the chronologically ordered raw data, the methodology first spatially sorts the measurements and then approximates them to a mean trajectory. The deviations between the single pose observations and the mean trajectory indicate the precision of the observed poses. With the addition of a higher-order reference, our methodology also determines the accuracy of the system under test. The applicability of our method is demonstrated by an exemplary evaluation of a low-cost inertial navigation system.

Improving detectability of auditory interfaces for medical alarms through temporal variation in amplitude envelope

BackgroundAuditory interfaces play a vital role in many applications, informing users about both urgent and routine information critical to safety. Unfortunately, problems related to high alarm rates, low reliability, and sound annoyance create barriers to optimising the quality of patient care in perioperative medicine and critical care. Here, we explore how to reduce annoyance and improve detection by manipulating a sound's temporal envelope or the way its energy changes over time. MethodsIn the first experiment, participants were asked to detect a series of percussive and flat tones presented at six signal-to-noise ratios while performing a concurrent speech comprehension task. In the second experiment, different participants were asked to evaluate the relative annoyance of these same sounds. ResultsRelative to industry-standard flat tones, percussive tones were significantly less annoying and more detectable. Although more detectable, percussive tones did not impair concurrent speech comprehension. ConclusionsTemporal variation in amplitude envelope represents a promising path towards improving auditory interfaces for patient monitoring. By using temporally variable sounds, auditory interfaces can be more effective in alerting users. This is important for safety-critical areas, such as medical alarms, where annoyance often limits efficacy. As this manipulation can preserve the pitch and rhythm of tone sequences, it is compatible with users' pre-existing knowledge of current alarms.

A Brief Comparison Between White Box, Targeted Adversarial Attacks in Deep Neural Networks

Today, neural networks are used in various domains, in most of which it is critical to have reliable and correct output. This is why adversarial attacks make deep neural networks less reliable to be used in safety-critical areas. Hence, it is important to study the potential attack methods to be able to develop much more robust networks. In this paper, we review four white box, targeted adversarial attacks, and compare them in terms of their misclassification rate, targeted misclassification rate, attack duration, and imperceptibility. Our goal is to find the attack(s), which would be efficient, generate adversarial samples with small perturbations, and be undetectable to the human eye.

A comprehensive empirical study on bug characteristics of deep learning frameworks

Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, such as self-driving cars and medical diagnostics, for DL frameworks, characterizing their bugs and thus helping researchers to design specific quality assurance techniques become desperately needed. Our research aims to characterize bugs typical of DL frameworks at the source code level for an in-depth analysis of bug symptoms, root causes, and bug fixes. In this way, we hope to provide insights for researchers to design automatic quality assurance techniques, such as automatic repair techniques and fault location techniques, applicable to DL frameworks and DL-based software systems. We started by summarizing the DL framework reference architecture and proposing the DL framework bug taxonomy. Then, we mined 1,127 DL framework bug reports from eight popular DL frameworks and labeled the bug types, root causes, and symptoms. Finally, we discussed the bug characteristics and explored how developers could possibly deal with these bugs. Our main findings are: (i) DNN model building bugs and general type bugs accounted for one-third of the total defects. (ii) DNN model building bugs are more prone to algorithm logic constraints, internal API errors, and data/numerical errors. (iii) Fifteen bug-fixing patterns are summarized, providing reference for common DL framework bug repair and future research on the development of automatic DL framework bug detection tools. By analyzing the bug-fixing changes, we characterize the occurrences, root causes, symptoms, and fixing of these bugs. The study results have provided researchers with insights into how to ensure DL framework quality and presented actionable suggestions for DL framework developers to improve their code quality.

Uncertainty Estimation Using Variational Mixture of Gaussians Capsule Network for Health Image Classification.

Capsule Networks have shown great promise in image recognition due to their ability to recognize the pose, texture, and deformation of objects and object parts. However, the majority of the existing capsule networks are deterministic with limited ability to express uncertainty. Many of them tend to be overconfident on out-of-distribution data, making them less trustworthy and hence reducing their suitability for practical adoption in safety-critical areas such as health and self-driving cars. In this work, we propose a capsule network based on a variational mixture of Gaussians to train distributions of network weights as opposed to a single set of weights and enable the model to express its predictive uncertainty on out-of-distribution data. Training distributions of weights have the added advantage of avoiding overfitting on smaller datasets which are common in health and other fields. Although Bayesian neural networks are known to exhibit slow training and convergence, experimental results show that the proposed model can retrieve only relevant features, converge faster, is less computationally complex, can effectively express its predictive uncertainties, and achieve performance values that are comparable to the state-of-the-art models. This is an indication that CapsNets can exhibit the transparency, credibility, reliability, and interpretability required for practical adoption.

TimeREISE: Time Series Randomized Evolving Input Sample Explanation.

Deep neural networks are one of the most successful classifiers across different domains. However, their use is limited in safety-critical areas due to their limitations concerning interpretability. The research field of explainable artificial intelligence addresses this problem. However, most interpretability methods align to the imaging modality by design. The paper introduces TimeREISE, a model agnostic attribution method that shows success in the context of time series classification. The method applies perturbations to the input and considers different attribution map characteristics such as the granularity and density of an attribution map. The approach demonstrates superior performance compared to existing methods concerning different well-established measurements. TimeREISE shows impressive results in the deletion and insertion test, Infidelity, and Sensitivity. Concerning the continuity of an explanation, it showed superior performance while preserving the correctness of the attribution map. Additional sanity checks prove the correctness of the approach and its dependency on the model parameters. TimeREISE scales well with an increasing number of channels and timesteps. TimeREISE applies to any time series classification network and does not rely on prior data knowledge. TimeREISE is suited for any usecase independent of dataset characteristics such as sequence length, channel number, and number of classes.

Navigation-Oriented Scene Understanding for Robotic Autonomy: Learning to Segment Driveability in Egocentric Images

This work tackles scene understanding for outdoor robotic navigation, solely relying on images captured by an on-board camera. Conventional visual scene understanding interprets the environment based on specific descriptive categories. However, such a representation is not directly interpretable for decision-making and constrains robot operation to a specific domain. Thus, we propose to segment egocentric images directly in terms of how a robot can navigate in them, and tailor the learning problem to an autonomous navigation task. Building around an image segmentation network, we present a generic affordance consisting of 3 driveability levels which can broadly apply to both urban and off-road scenes. By encoding these levels with soft ordinal labels, we incorporate inter-class distances during learning which improves segmentation compared to standard “hard” one-hot labelling. In addition, we propose a navigation-oriented pixel-wise loss weighting method which assigns higher importance to safety-critical areas. We evaluate our approach on large-scale public image segmentation datasets ranging from sunny city streets to snowy forest trails. In a cross-dataset generalization experiment, we show that our affordance learning scheme can be applied across a diverse mix of datasets and improves driveability estimation in unseen environments compared to general-purpose, single-dataset segmentation.

Experimental and numerical investigation of Inconel 718 machining with worn tools

Nickel based superalloy is widely used in safety critical areas and its machined surface integrity is considered as one of the most important aspects regarding their work performance. Due to the reality that most of the machined surface is actually generated with tools in the worn condition because of the rapid worn when machining these kinds of materials, it is essential to understand the machining performance with worn tools. In the present investigation, both numerical and experimental methods are employed to study the cutting process when machining Inconel 718 with worn tools, including chip formation, force variation, tool wear and thermal distribution, as well as the acquired subsurface microstructure and mechanical characteristics. The results clearly shows the influence from wear tool at difference cutting speeds on the thermal distribution, force variation and tool flank wear performance. Obvious white layer appeared on the subsurface is believed because of the existing of the tool wear which causes the intense of friction in the tool-surface contact and significant heat generation changing the stress condition and thermal distribution. The EBSD and SEM results also clearly present the subsurface material alternation and deformation difference in various cutting conditions. The similar findings acquired from the numerical results not only shows good coherence with the experimental study, but also emphasize the ability to investigate the cutting performance with worn tools in numerical way, which can reduce the needed resource and study the deformation in generated surface properties rapidly.