Safety-critical Research Articles

Using Approximate Circuits Against Hardware Trojans

Hardware Trojans (HTs) are malicious alterations in Integrated Circuits (ICs) that pose an important threat to safety-critical systems. Many techniques based on logic testing or side-channel analysis have been proposed in the literature aiming at detecting such malicious modifications in fabricated ICs. The detection of HTs is becoming more challenging with the shrinking of the technology and the impact of process variations. Therefore there is a need to neutralize the effect of HTs when the typical detection mechanism fails. In this work, we propose to tackle the effect of HTs by leveraging fault-tolerant techniques like Triple Modular Redundancy (TMR). More specifically, we propose to use an approximate TMR in order to neutralize the effects of HTs while saving area and increasing the complexity of inserting HTs. The efficiency of the proposed approach has been evaluated by using the ISCAS’85 benchmarks and stealthy ad-hoc HTs.

A Safety-Critical Decision-Making and Control Framework Combining Machine-Learning-Based and Rule-Based Algorithms

<div>While machine-learning-based methods suffer from a lack of transparency, rule-based (RB) methods dominate safety-critical systems. Yet the RB approaches cannot compete with the first ones in robustness to multiple system requirements, for instance, simultaneously addressing safety, comfort, and efficiency. Hence, this article proposes a decision-making and control framework which profits from the advantages of both the RB and machine-learning-based techniques while compensating for their disadvantages. The proposed method embodies two controllers operating in parallel, called Safety and Learned. An RB switching logic selects one of the actions transmitted from both controllers. The Safety controller is prioritized whenever the Learned one does not meet the safety constraint, and also directly participates in the Learned controller training. Decision-making and control in autonomous driving are chosen as the system case study, where an autonomous vehicle (AV) learns a multitask policy to safely execute an unprotected left turn. Multiple requirements (i.e., safety, efficiency, and comfort) are set to vehicle motion. A numerical simulation is performed for the proposed framework validation, where its ability to satisfy the requirements and robustness to changing environments is successfully demonstrated.</div>

Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-based Safety-critical Systems

When Deep Neural Networks (DNNs) are used in safety-critical systems, engineers should determine the safety risks associated with failures (i.e., erroneous outputs) observed during testing. For DNNs processing images, engineers visually inspect all failure-inducing images to determine common characteristics among them. Such characteristics correspond to hazard-triggering events (e.g., low illumination) that are essential inputs for safety analysis. Though informative, such activity is expensive and error prone. To support such safety analysis practices, we propose Simulator-based Explanations for DNN failurEs (SEDE), a technique that generates readable descriptions for commonalities in failure-inducing, real-world images and improves the DNN through effective retraining. SEDE leverages the availability of simulators, which are commonly used for cyber-physical systems. It relies on genetic algorithms to drive simulators toward the generation of images that are similar to failure-inducing, real-world images in the test set; it then employs rule learning algorithms to derive expressions that capture commonalities in terms of simulator parameter values. The derived expressions are then used to generate additional images to retrain and improve the DNN. With DNNs performing in-car sensing tasks, SEDE successfully characterized hazard-triggering events leading to a DNN accuracy drop. Also, SEDE enabled retraining leading to significant improvements in DNN accuracy, up to 18 percentage points.

Testing Feedforward Neural Networks Training Programs

At present, we are witnessing an increasing effort to improve the performance and trustworthiness of Deep Neural Networks (DNNs), with the aim to enable their adoption in safety critical systems such as self-driving cars or aircraft collision-avoidance systems. Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of DNN models. These techniques assume implicitly that the training program is bug-free and appropriately configured. However, satisfying this assumption for a novel problem requires significant engineering work to prepare the data, design the DNN, implement the training program, and tune the hyperparameters to produce the model for which current automated test data generators search for corner-case behaviors. All these model training steps can be error prone. Therefore, it is crucial to detect and correct errors throughout all the engineering steps of DNN-based software systems and not only on the resulting DNN model. In this article, we gather a catalog of training issues and based on their symptoms and their effects on the behavior of the training program, we propose practical verification routines to detect the aforementioned issues, automatically, by continuously validating that some important properties of the learning dynamics hold during the training. Then, we design TheDeepChecker , an end-to-end property-based debugging approach for DNN training programs and implement it as a TensorFlow-based library. As an empirical evaluation, we conduct a case study to assess the effectiveness of TheDeepChecker on synthetic and real-world buggy DL programs and compare its performance to that of the Amazon SageMaker Debugger ( SMD ). Results show that TheDeepChecker ’s on-execution validation of DNN-based program’s properties through three sequential phases (pre-, on-, and post-fitting) succeeds in revealing several coding bugs and system misconfigurations errors early on and at a low cost. Moreover, our property-based approach outperforms the SMD ’s offline rules verification on training logs in terms of detection accuracy for unstable learning issues and coverage of additional DL bugs.

Neural Network-based Tool for Survivability Assessment of K-variant Systems

The K-variant is a multi-variant architecture to enhance the security of the time-bounded mission and safety-critical systems. Variants in the K-variant architecture are generated by controlled source program transformations. Previous experimental studies showed that the K-variant architecture might improve the security of systems against memory exploitation attacks. In order to estimate the survivability of K-variant systems, simulation techniques are utilized. However, these techniques are slow and may not be practical for the design of K-variant systems. Therefore, fast and highly accurate estimations of the survivability of K-variant systems are necessary for developers. The neural networks may allow quick and accurate estimation of the survivability of K-variant systems. The developed neural network-based tool can make quick and precise estimations of the survivability of K-variant systems under different conditions. In this paper, the accuracy of the neural network-based tool is investigated in an experimental study. The neural network-based tool estimations are compared with a K-variant attack emulator in three programs for up to ten variant systems under four attack types and three attack durations. The experimental study demonstrates that the neural network-based tool makes fast and accurate estimations of the survivability of K-variant systems under all the conditions investigated.

Coupling and complexity at the global scale: flows, networks, interconnectedness and synchronicity (e.g. Covid-19)

The argument of this paper is that the notions of coupling and complexity introduced by Perrow to discuss high-risk technologies need to be scaled up from their original scope to a wider, global one considering the evolution of the operational landscape of safety–critical systems. One value of Perrow in the 1980 s was to trigger core debates, contributing to a new form of causal reasoning. It provided a way of thinking about causality within a context of expanding sociotechnical systems such as nuclear power plants, civil commercial aviation or petroleum infrastructures. In the 2020 s, high-risk technologies do not only represent potential for single catastrophic events anymore but are also the sources of wider problems associated with globalised flows (systemic risks) and ecological degradation known as anthropocene (leading to existential risks). In this context, the scope, scale and timeframe of high-risk systems is extended in comparison to Perrow’s original work. In these new circumstances, coupling and complexity categories apply to a much wider spectrum of issues than initially conceived by Perrow and notions such as flows, nodes, networks, interconnectedness, and synchronicity should be incorporated to illustrate a move to the global stage. After a presentation and discussion of Perrow’s coupling and interaction, the “global turn” in research is discussed, using sociology and history as examples. It is followed by the contribution by Guillén on the “architecture of collapse” characterising our contemporary era. Its implications for safety science research are then explored using Covid-19 as an example.

Reliability and failure modelling of microelectronic packages based on ultrasonic nondestructive evaluation data

Reliability testing and failure modelling is crucial and very challenging for modern electronics, especially safety critical electronics-based systems working under harsh environmental conditions. In this paper, an approach based on ultrasonic Non-Destructive Evaluation (NDE) is proposed to establish a failure model for solder joints which is urgently needed for prognostic and health management of electronics. Printed Circuit Boards (PCB) containing flip-chip and Area Array packages were designed and aged using Accelerated Thermal Cycling (ATC) testing. During ATC testing solder joint degradation was regularly monitored by taking out the test boards from the temperature chamber at four thermal cycle intervals for ultrasonic micro-imaging. Data pre-processing techniques including dynamic range-based intensity normalization, gain compensation, and calibration for random transducer defocusing errors were developed to enhance the data consistency, integrity and accuracy. Furthermore, solder joint image labelling and segmentation methods were developed by exploiting the geometrical features of the solder joints to extract individual solder joints and Region of Interests (ROIs) from the sequential ultrasound images collected throughout the testing. The mean intensity of the ROIs was used as the precursor to degradation to build the solder joint failure model, from which cycles to failure for individual solder joints are finally derived for life prediction.

Identifying and managing data quality requirements: a design science study in the field of automated driving

Good data quality is crucial for any data-driven system’s effective and safe operation. For critical safety systems, the significance of data quality is even higher since incorrect or low-quality data may cause fatal faults. However, there are challenges in identifying and managing data quality. In particular, there is no accepted process to define and continuously test data quality concerning what is necessary for operating the system. This lack is problematic because even safety-critical systems become increasingly dependent on data. Here, we propose a Candidate Framework for Data Quality Assessment and Maintenance (CaFDaQAM) to systematically manage data quality and related requirements based on design science research. The framework is constructed based on an advanced driver assistance system (ADAS) case study. The study is based on empirical data from a literature review, focus groups, and design workshops. The proposed framework consists of four components: a Data Quality Workflow, a List of Data Quality Challenges, a List of Data Quality Attributes, and Solution Candidates. Together, the components act as tools for data quality assessment and maintenance. The candidate framework and its components were validated in a focus group.

Woodside offshore dry deluge testing

Wet deluge testing of firewater systems is considered a recognised contributing factor to accelerated corrosion on offshore facilities. Spraying sensitive equipment and structures with sea water is not only disruptive to the facility, it can also cause process upsets and is labour intensive with set-up and subsequent fresh water washdown. While ensuring the performance of this safety-critical system is vital through regular testing, adopting new methods with less adverse impacts is of high importance. Dry deluge testing using a propylene glycol-based vapour as an alternate testing medium has been adopted on offshore facilities in the US, Dutch, Danish and UK sectors, significantly reducing the frequency of wet testing. A generator is used to produce smoke-like vapour, which is then pumped into the system piping through existing test connection points while the system remains online. The vapour migrates throughout the distribution piping, ultimately exiting the deluge nozzles to visually identify any blockages. This method of testing will reduce costs to businesses, not only for deluge testing itself, but more importantly from an asset fabric maintenance perspective. Woodside Energy successfully carried out the first offshore dry deluge testing in Australia on the Okha FPSO (floating production, storage and offloading) in 2021. The testing covered varied deluge and foam system arrangements with different nozzle types. This paper details the dry deluge tests conducted on Okha, the results gathered, main operational and implementation concerns of the testing and recommendations on further collaboration and improvements for adopting this test method across the Australian oil and gas industry.

A fault-tolerant resource locking protocol for multiprocessor real-time systems

This paper presents the first fault-tolerant resource locking protocol for multiprocessor real-time systems. Resource locking protocols control access to shared resources in real-time systems. Most of the previous studies in resource locking protocols have focused on decreasing blocking time and resource conflicts. The use of such protocols in multiprocessor safety-critical real-time systems necessitates fault tolerance in the design of these systems. However, in research focused on multiprocessor real-time systems, fault tolerance has received low attention in resource allocation studies. In addition, many solutions proposed to improve the reliability in real-time scheduling ignored resource models in their task model. This paper proposes a novel fault-tolerant resource locking protocol, called FTPIA-NPCS, which considers transient faults in shared resources. FTPIA-NPCS proposes a checkpointing-based fault-tolerant mechanism to recover from errors caused by transient faults. In this mechanism, a checkpoint is taken from the tasks, before each resource allocation. In addition to providing fault tolerance, FTPIA-NPCS solves the problem of priority inversion in NPCS protocol for periodic tasks with hard deadlines. The simulation results demonstrate that FTPIA-NPCS tolerates at least one transient fault in resources for each execution of the periodic tasks with 15% blocking time overhead in comparison to its non-fault-tolerant configuration, called PIA-NPCS. Furthermore, the blocking time of higher-priority tasks is minimized in comparison to PIA-NPCS. An analytical evaluation of the protocol supports the simulation results.

Mission abort strategy for generalized k-out-of-n: F systems considering competing failure criteria

For safety-critical systems such as submarines and solar lighting system, mission abort is an effective way to enhance system survivability when a certain malfunction condition is met. This paper contributes by presenting a bivariate mission abort policy for generalized k-out-of- n: F systems that fail if there are at least m non-overlapping kc consecutive failed components or at least kt failed components. When the number of non-overlapping kc consecutive failed components reaches a preset level or the total number of failed components exceeds a predetermined value, the mission is aborted, and then a rescue procedure is initiated. Mission reliability and system survivability are derived by employing a two-step finite Markov chain imbedding approach. The optimization models are formulated with the purpose of maximizing the mission reliability, and minimizing the expected total cost of mission failure and system failure, respectively. A numerical example based on a solar lighting system is presented to illustrate the applicability of the proposed policies.

Hybrid DFM – Petri net approach for dynamic reliability analysis of smart transmitters

AbstractSmart transmitters are increasingly being deployed in safety critical systems. Although the processing and diagnostic features provided by digital systems are advantageous, it introduces additional complexity in reliability modelling of systems. The conventional Fault Tree Analysis (FTA) method for system reliability analysis has its shortcomings in modelling digital systems, to incorporate its dynamic system interaction and failure modes in detail. This is because some of the failure modes may arise due to the dynamic interactions between the modules of the system. Dynamic Flowgraph Methodology (DFM) was identified as one of the optimistic approaches to handle these interactions. DFM approach combines multi‐valued logic modelling and analysis capabilities to handle systems consisting of components that have multiple degraded states and exhibit dynamic behaviour. However, only few DFM tools have been developed and some of them are not available in public domain. The DFM approach relies heavily on development and implementation of its decision tables. The logic information present within the decision tables can be meticulously implemented using Petri Nets. Also, there are plenty of free Petri Nets tools available online. Thus, the idea of DFM was used in this work to capture the dynamic interactions by developing decision tables. The decision tables were transformed into the Petri Net models to perform the analysis. The paper demonstrates the potential of the hybrid DFM – Petri Net approach for dynamic reliability analysis of smart transmitter.

Black-Box Testing of Deep Neural Networks through Test Case Diversity

Deep Neural Networks (DNNs) have been extensively used in many areas including image processing, medical diagnostics, and autonomous driving. However, DNNs can exhibit erroneous behaviours that may lead to critical errors, especially when used in safety-critical systems. Inspired by testing techniques for traditional software systems, researchers have proposed neuron coverage criteria, as an analogy to source code coverage, to guide the testing of DNN models. Despite very active research on DNN coverage, several recent studies have questioned the usefulness of such criteria in guiding DNN testing. Further, from a practical standpoint, these criteria are white-box as they require access to the internals or training data of DNN models, which is in many contexts not feasible or convenient. In this paper, we investigate black-box input diversity metrics as an alternative to white-box coverage criteria. To this end, we first select and adapt three diversity metrics and study, in a controlled manner, their capacity to measure actual diversity in input sets. We then analyse their statistical association with fault detection using four datasets and five DNN models. We further compare diversity with state-of-the-art white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria to effectively guide the testing of DNNs. Indeed, we found that one of our selected black-box diversity metrics far outperforms existing coverage criteria in terms of fault-revealing capability and computational time. Results also confirm the suspicions that state-of-the-art coverage metrics are not adequate to guide the construction of test input sets to detect as many faults as possible with natural inputs.

On Neural Networks Redundancy and Diversity for Their Use in Safety-Critical Systems

An increasing number of critical functionalities integrated in embedded critical systems rely on deep learning (DL) technology. This article summarizes certain key aspects of DL’s intrinsic stochastic and training-data-dependent nature that are at odds with current domain-specific functional safety standards. We exemplify how redundancy and diversity of neural networks can help to reconcile DL technology and functional safety requirements.

Performance Evaluation and Comparison of Three-Phase and Six-Phase Winding in Ultrahigh-Speed Machine for High-Power Application

This article investigates the influence of multiphase winding topologies in high-power ultra-high-speed machines (HP-UHSM) of 500 kr&#x002F;min. At this speed level, increasing the rotor&#x0027;s magnetic loading excites its critical bending resonances and leads to structural breakdown. On the other hand, increasing the stator&#x0027;s electric loading using the three-phase winding increases unwanted vibrations in a slotted stator and reduces the electromagnetic inter-action of the stator and rotor in a slotless stator. Consequently, the maximum output power level of UHSM (500 kr&#x002F;min or more) is limited to a few hundred watts only in the state-of-the-art. To over-come such a critical limitation, this article proposes a new design methodology for HP-UHSM, where the rotor&#x0027;s bending resonances and centrifugal stresses are restricted by limiting the maximum aspect ratio (<i>L&#x002F;D</i>), and an optimal multiphase winding is adopted in the slotless stator to increase the power level by effective electric loading. Also, a Multiphysics optimization is utilized to obtain the optimum magnetic loading and electric loading, where the bending resonance and other system limits are defined using multi-disciplinary design constraints. It is observed that the multiphase winding provides an added degree of freedom to increase the power level of UHSM without exciting the rotor&#x0027;s bending resonances and structural breakdown. Using the proposed method, a multiphase 2 kW 500 kr&#x002F;min HP-UHSM has been designed for the safety-critical AMEBA system and compared its Multiphysics performance with the three-phase machine having the same volume. Finally, exten-sive experiments are performed on both prototypes to validate the effectiveness of the proposed method. It is shown that the multi-phase HP-UHSM has no critical bending resonance below the 500 kr&#x002F;min, and it has 16.3&#x0025; higher output power with 1.18&#x0025; higher efficiency and 28.6&#x0025; lower back-EMF than the three-phase design.

Security risks in the software development lifecycle: A review

Software security is one of the most critical concerns in modern software development, especially in safety-critical systems whose failure can lead to environmental damage, substantial property, or loss of human lives. In addition, flawed applications have been shown to exhibit unpredictable behavior while software products with numerous vulnerabilities present attack vectors that can be exploited by attackers. To address some of these problems, vulnerability prediction has been deployed for early detection of security risks in the software development lifecycle (SDLC). This can potentially facilitate decision making during the SDLC, resulting in the production of more secure software. Prioritizing security during SDLC permits developers and stakeholders to identify and resolve possible security concerns early on in the process. The aim of this paper is therefore to offer some in-depth review of software systems security issues. In addition, the various measures that have been put in place to mitigate security issues during SDLC are discussed.

Safety-Critical Control for Control Affine Systems under Spatio-Temporal and Input Constraints

Safety-critical control is a type of modern control task where potentially conflicting stability, safety, and input constraints coexist. In this paper, the Prescribed-Time Zeroing Control Barrier Function (PT-ZCBF) is introduced, which can be applied as a prescribed-time stability constraint in safety-critical control tasks. Furthermore, we formulate a PT-ZCBF-based Quadratic Program (QP), which is able to mediate the potentially conflicting constraints of safety-critical control. The solution of the newly designed QP, acting as the control input of a safety-critical system, can drive the closed-loop trajectories to converge in a user-defined prescribed time period while observing the safety and input constraints. Finally, we use the Adaptive Cruise Control (ACC) problem as an example of numerical simulation to evaluate the performance of the QP-based method.

ProCon: An automated process-centric quality constraints checking framework

When dealing with safety–critical systems, various regulations, standards, and guidelines stipulate stringent requirements for certification and traceability of artifacts, but typically lack details with regards to the corresponding software engineering process. Given the industrial practice of only using semi-formal notations for describing engineering processes – with the lack of proper tool mapping – engineers and developers need to invest a significant amount of time and effort to ensure that all steps mandated by quality assurance are followed. The sheer size and complexity of systems and regulations make manual, timely feedback from Quality Assurance (QA) engineers infeasible. In order to address these issues, in this paper, we propose a novel framework for tracking, and “passively” executing processes in the background, automatically checking QA constraints depending on process progress, and informing the developer of unfulfilled QA constraints. We evaluate our approach by applying it to three case studies: a safety–critical open-source community system, a safety–critical system in the air-traffic control domain, and a non-safety–critical, web-based system. Results from our analysis confirm that trace links are often corrected or completed after the work step has been considered finished, and the engineer has already moved on to another step. Thus, support for timely and automated constraint checking has significant potential to reduce rework as the engineer receives continuous feedback already during their work step.

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

Deep neural networks (DNNs) have demonstrated superior performance over classical machine learning to support many features in safety-critical systems. Although DNNs are now widely used in such systems (e.g., self driving cars), there is limited progress regarding automated support for functional safety analysis in DNN-based systems. For example, the identification of root causes of errors, to enable both risk analysis and DNN retraining, remains an open problem. In this article, we propose SAFE, a black-box approach to automatically characterize the root causes of DNN errors. SAFE relies on a transfer learning model pre-trained on ImageNet to extract the features from error-inducing images. It then applies a density-based clustering algorithm to detect arbitrary shaped clusters of images modeling plausible causes of error. Last, clusters are used to effectively retrain and improve the DNN. The black-box nature of SAFE is motivated by our objective not to require changes or even access to the DNN internals to facilitate adoption. Experimental results show the superior ability of SAFE in identifying different root causes of DNN errors based on case studies in the automotive domain. It also yields significant improvements in DNN accuracy after retraining, while saving significant execution time and memory when compared to alternatives.

Risk-informed collision avoidance system design for maritime autonomous surface ships

The maritime industry is paving the way towards developing Maritime Autonomous Surface Ships (MASSs) through the adoption of key enabling technologies for safety-critical operations, which are associated with new challenges, especially at their early design phase. This study aims to develop a methodology to conduct the risk-informed design for the Collision Avoidance System (CAS) of MASSs. Pertinent regulatory instruments are reviewed to identify functional and system requirements and develop a baseline CAS configuration at the component level. Quantitative Fault Tree Analysis is performed to derive risk metrics, such as probability of failure, Importance measures, and Minimal Cut Sets, whereas criticality analysis is conducted to recommend risk-reducing measures. A Short Sea Shipping case study is investigated considering four operating modes based on various weather and illumination conditions. Results demonstrate that the developed Fault Tree diagram provides a robust representation of the CAS failure. The most critical components are found to be related to the Intention Communication and Situation Awareness Systems, the redundancy of which leads to 91% reduction of the CAS probability of failure. This study contributes towards the risk-informed design of safety-critical systems required for the development of MASSs.