Safety-critical Real-time Systems Research Articles

A fault-tolerant resource locking protocol for multiprocessor real-time systems

This paper presents the first fault-tolerant resource locking protocol for multiprocessor real-time systems. Resource locking protocols control access to shared resources in real-time systems. Most of the previous studies in resource locking protocols have focused on decreasing blocking time and resource conflicts. The use of such protocols in multiprocessor safety-critical real-time systems necessitates fault tolerance in the design of these systems. However, in research focused on multiprocessor real-time systems, fault tolerance has received low attention in resource allocation studies. In addition, many solutions proposed to improve the reliability in real-time scheduling ignored resource models in their task model. This paper proposes a novel fault-tolerant resource locking protocol, called FTPIA-NPCS, which considers transient faults in shared resources. FTPIA-NPCS proposes a checkpointing-based fault-tolerant mechanism to recover from errors caused by transient faults. In this mechanism, a checkpoint is taken from the tasks, before each resource allocation. In addition to providing fault tolerance, FTPIA-NPCS solves the problem of priority inversion in NPCS protocol for periodic tasks with hard deadlines. The simulation results demonstrate that FTPIA-NPCS tolerates at least one transient fault in resources for each execution of the periodic tasks with 15% blocking time overhead in comparison to its non-fault-tolerant configuration, called PIA-NPCS. Furthermore, the blocking time of higher-priority tasks is minimized in comparison to PIA-NPCS. An analytical evaluation of the protocol supports the simulation results.

A heuristic mixed real-time task allocation of virtual utilization in multi-core processor

Multi-core processor is widely used as the running platform for safety-critical real-time systems such as spacecraft, and various types of real-time tasks are dynamically added at runtime. In order to improve the utilization of multi-core processors and ensure the real-time performance of the system, it is necessary to adopt a reasonable real-time task allocation method, but the existing methods are only for single-core processors or the performance is too low to be applicable. Aiming at the task allocation problem when mixed real-time tasks are dynamically added, we propose a heuristic mixed real-time task allocation algorithm of virtual utilization VU-WF (Virtual Utilization Worst Fit) in multi-core processor. First, a 4-tuple task model is established to describe the fixed-point task and the sporadic task in a unified manner. Then, a VDS (Virtual Deferral Server) for serving execution requests of fixed-point task is constructed and a schedulability test of the mixed task set is derived. Finally, combined with the analysis of VDS's capacity, VU-WF is proposed, which selects cores in ascending order of virtual utilization for the schedulability test. Experiments show that the overall performance of VU-WF is better than available algorithms, not only has a good schedulable ratio and load balancing but also has the lowest runtime overhead. In a 4-core processor, compared with available algorithms of the same schedulability ratio, the load balancing is improved by 73.9%, and the runtime overhead is reduced by 38.3%. In addition, we also develop a visual multi-core mixed task scheduling simulator RT-MCSS (open source) to facilitate the design and verification of multi-core scheduling for users. As the high performance, VU-WF can be widely used in resource-constrained and safety-critical real-time systems, such as spacecraft, self-driving cars, industrial robots, etc.

FATS-2TC: A Fault Tolerant real-time Scheduler for energy and temperature aware heterogeneous platforms with Two types of Cores

Fault tolerance in safety-critical real-time systems is indispensable. With the increasing density of transistors on MPSoCs, energy consumption and peak temperatures are exponentially increasing, thus making the method unreliable. This paper devises a heuristic scheduling strategy called FATS-2TC, which solves the issue of the simultaneous control of energy and peak temperature levels using the standby-sparing mechanism on systems with two types of cores (like big.LITTLE architecture) to make it resistant to transient faults. Our experimental results demonstrate that FATS-2TC can save considerable energy and lower core peak temperatures than a state-of-the-art energy-efficient fault-tolerant scheduler.

A Federated Deep Learning Empowered Resource Management Method to Optimize 5G and 6G Quality of Services (QoS)

The quality of service (QoS) in 5G/6G communication enormously depends upon the mobility and agility of the network architecture. An increase in the possible uses of 5G vehicular network simultaneously expands the scope of the network’s quality of service (QoS). To this end, a safety-critical real-time system has become one of the most demanding criteria for the vehicular network. Although different mathematical and computation methods have traditionally been used to optimize the allocation of resources, but the nonconvexity of optimization issues creates unique type of challenges. In recent years, machine learning (ML) has emerged as a valuable tool for dealing with computational complexity that involves large amounts of data in heterogeneous vehicular networks. By using optimization and cutting-edge machine learning techniques, this article gives readers an insight about how 5G vehicular network resources can be allocated to reinforce network communication. Furthermore, a new federated deep reinforcement learning- (FDRL-) based vehicle communication method is presented as a new insight. Finally, a UAV-aided vehicular communication system based on FDRL-based UAVs is proposed as a novel resource management technique to optimize 5G and 6G quality of services.

A Flexible Communication Protocol With Guaranteed Determinism for Distributed, Safety-Critical Real-Time Systems

Dependable, safety-critical real-time (SCRT) systems are becoming increasingly important and complex. Examples of such systems are autonomous or self-driving cars which are poised to revolutionise the transport industry. A critical part of these SCRT systems is the network communication protocol that is used by components in an SCRT system to exchange data. Communication protocols for SCRT systems are required to exhibit predictable, worst-case execution times and thus have to be designed in a more static and less flexible way. To ensure this predictability, current state-of-art communication protocols for SCRT systems are based on the Time-Triggered Architecture (TTA), where static and equal-length time-slots are used for all nodes to access the communication channel, irrespective of the size of their transmission payload. This determinism forms the basis of predictable timing, behaviour and fault tolerance. However, this determinism comes at the cost of poor channel and bandwidth utilisation, which hinders the development of SCRT systems. In this paper, we propose a more flexible approach, INCUS+, that allocates the slot length of a node based on its transmission requirements in a Time Division Multiple Access (TDMA) round. We achieve this while retaining the level of dependability required for SCRT systems and ensuring fail-silence. We validate this through formal verification of the timing parameters for the transmission windows of all participating nodes as well as independent bus guardians. Our design exhibits a significant improvement in bandwidth and channel utilisation, as we demonstrate in an autonomous vehicle case study.

Cross-Layer Adaptation with Safety-Assured Proactive Task Job Skipping

During the operation of many real-time safety-critical systems, there are often strong needs for adapting to a dynamic environment or evolving mission objectives, e.g., increasing sampling and control frequencies of some functions to improve their performance under certain situations. However, a system's ability to adapt is often limited by tight resource constraints and rigid periodic execution requirements. In this work, we present a cross-layer approach to improve system adaptability by allowing proactive skipping of task executions, so that the resources can be either saved directly or re-allocated to other tasks for their performance improvement. Our approach includes three novel elements: (1) formal methods for deriving the feasible skipping choices of control tasks with safety guarantees at the functional layer, (2) a schedulability analysis method for assessing system feasibility at the architectural layer under allowed task job skippings, and (3) a runtime adaptation algorithm that efficiently explores job skipping choices and task priorities for meeting system adaptation requirements while ensuring system safety and timing correctness. Experiments demonstrate the effectiveness of our approach in meeting system adaptation needs.

Investigating data representation for efficient and reliable Convolutional Neural Networks

Nowadays, Convolutional Neural Networks (CNNs) are widely used as prediction models in different fields, with intensive use in real-time safety-critical systems. Recent studies have demonstrated that hardware faults induced by an external perturbation or aging effects, may significantly impact the CNN inference, leading to prediction failures. Therefore, ensuring the reliability of CNN platforms is crucial, especially when deployed in critical applications. A lot of effort has been made to reduce the memory and energy footprint of CNNs, paving the way to the adoption of approximate computing techniques such as quantization, reduced precision, weight sharing, and pruning. Unfortunately, approximate computing reduces the intrinsic redundancy of CNNs making them more efficient but less resilient to hardware faults. The goal of this work is twofold. First, we assess the reliability of a CNN when reduced bit widths and two different data types (floating- and fixed-point) are used to represent the network parameters (i.e., synaptic weights). Second, we intend to investigate the best compromise between data type, bit-widths reduction, and reliability. The characterization is performed through a fault injection environment built on the darknet open-source framework and targets two CNNs: LeNet-5 and YOLO. Experimental results show that fixed-point data provide the best trade-off between memory footprint reduction and CNN resilience. In particular, for LeNet-5, we achieved a 4X memory footprint reduction at the cost of a slightly reduced reliability (0.45% of critical faults) without retraining the CNN.

Generation of a fault-tolerant clock through redundant crystal oscillators

Having a precise and stable clock that is still fault tolerant is a fundamental prerequisite in safety critical real-time systems. However, combining redundant independent clock sources to form a unified fault-tolerant clock supply is non-trivial, especially when redundant clock outputs are required – e.g., for supplying the replicated nodes within a TMR architecture through a clock network that does not suffer from a single point of failure. Having these outputs fail independent but still keeping them tightly synchronized is highly desirable, as it substantially eases the design of the overall architecture.In this paper we address exactly this challenge. Our approach extends an existing, ring-oscillator like distributed clock generation scheme by augmenting each of its constituent nodes with a stable clock reference. We introduce the appropriately modified algorithm and illustrate its operation by simulation experiments. These experiments further demonstrate that the four clock outputs of our circuit do not share a single point of failure, have small and bounded skew, remain stabilized to one crystal source during normal operation, do not propagate glitches from one failed clock to a correct one, and only exhibit slightly extended clock cycles during a short stabilization period after a component failure. In addition we give a rigorous formal proof for the correctness of the algorithm on an abstraction level that is close to the implementation.

Verification of concurrent code from synchronous specifications

• Methodology to generate concurrent code automatically from SIGNAL specifications. • New clock calculus method to explore the clock relations among signals. • Verification of concurrent code from SIGNAL specifications. The synchronous language SIGNAL is a formal specification formalism for developing safety-critical real-time systems. It is a multi-clocked data-flow modeling language suitable for specifying deterministic concurrent behaviors. Its model of computation and communication very well matches recent trends to utilize multi-core processors for executing real-time systems, by taking advantage of its concurrent semantics. The SIGNAL compiler generates code from data-flow specifications while analyzing and verifying safety properties of the system under design: deadlock-freedom, determinism. However, most of recent works have focused on generating sequential code from SIGNAL. Choosing the parallel library OpenMP as the target, this paper proposes a methodology to generate and verify concurrent code automatically from SIGNAL specifications. This is done by first exploring clock relations among signals by application of a so-called clock calculus. Then, specifications are translated into EDGs (Equation-Dependency Graphs) to analyze global data-dependency relations. An EDG is then partitioned into concurrent tasks to help explore parallelism in the original specification while preserving its semantic. Combined with clock relations, parallel tasks are finally mapped onto the OpenMP structures. The proposed approach is illustrated by a realistic case study.

A Hierarchical State Machine Model for Hazard Analysis of Real-time Safety Critical Systems

Real-time systems must avoid hazardous situations. To achieve this, their functionality should be investigated under time constraints. In this paper, a modeling based on Hierarchical Communicating Real-time State Machine (H- CRSM) and analysis methodology is proposed to examine statically ANSI-C code to obtain the hazardous events in the input system. A hazardous event equation is taken as input to the proposed system. The output is a list of hazardous scenarios. A path in the code showing the cause of the undesirable event is associated with each hazardous scenario. The strength of the proposed methodology is that the process of hazardous situations detection does not require running the ANSI-C program multiple times with different input values. It also focuses on analyzing the software level of the life cycle. Most of the verification tools check the system level. The system level may be bug-free but the software level may not.

Mixed-Criticality Scheduling Upon Permitted Failure Probability and Dynamic Priority

Many safety-critical real-time systems are considered certified when they meet failure probability requirements with respect to the maximum permitted incidences of failure per hour. In this article, the mixed-criticality task model with multiple worst case execution time (WCET) estimations is extended to incorporate such system-level certification restrictions. A new parameter is added to each task, characterizing the distribution of WCET estimations—the likelihood of all jobs of a task finishing their executions within the less pessimistic WCET estimates. Efficient algorithms are derived for scheduling mixed-criticality systems represented using this model for both uniprocessor and multiprocessor platforms for independent tasks. Furthermore, a 0/1 covariance matrix is introduced to represent the failure dependency between tasks. An efficient algorithm is proposed to schedule such failure-dependent tasks. Experimental analyses show our new model and algorithm outperform current state-of-the-art mixed-criticality scheduling algorithms.

Specification-Guided Automated Debugging of CPS Models

Simulink/Stateflow is the de facto tool for developing software for safety-critical real-time cyber-physical systems (CPSs). In Simulink, the model of a CPS is captured in a block diagram-based language, the model is simulated using the associated simulators and then the software code is generated automatically for the embedded controller. The presence of a bug in the Simulink model may lead to catastrophic failure during the execution of the system developed based on the model. Unlike in application software, finding bugs in Simulink models is challenging due to the hybrid nature of the model. We present an automated debugging methodology of a CPS model captured in Simulink. Our methodology has two main components-bug localization and model repair. For bug localization, we capture the requirements of the system in signal temporal logic (STL) and employ the runtime monitoring technique to generate a trace that violates the specification. The violating trace is used to identify the internal signals that have the potential to contribute to the violation. For precise bug localization by narrowing down the offending signals, we employ a matrix decomposition technique to find the signals contributing to the bug accurately. Our bug localization technique is precise enough to enable us to repair the model. If the bug is due to an inappropriate value for a model parameter, we employ a parameter tuning method to find a value for the parameter that repairs the model automatically. We carry out numerous case studies on Simulink models obtained from different sources and demonstrate that our automated debugging technology can fix the bugs in the Simulink models effectively.

Raising the Bar: Assurance Cases for Scientific Software

Assurance cases provide an organized and explicit argument for correctness. They can dramatically improve the quality of scientific software. Assurance cases are already effectively used for real-time safety critical systems. Their advantages for scientific software include engaging domain experts, producing only necessary documentation, and providing evidence that can be verified/replicated. This article illustrates assurance cases through the correctness case for 3dfim+, software for analyzing activity in the brain. The example justifies the value of assurance cases for scientific software, since the existing documentation is shown to have ambiguities and omissions, such as an incompletely defined ranking function and missing details on the coordinate system. We identified a serious concern for 3dfim+: running the software does not produce any warning about the necessity of using data that matches the parametric statistical model employed for the correlation calculations. Raising the bar for scientific software is both feasible and necessary.

Formal Verification of AADL Models by Event-B

AADL is widely used to depict the architecture and behavior of real-time safety-critical systems such as avionics and aerospace. The development of these systems has strict requirements for building fault-free systems. Formal verification is frequently applied to verify the critical properties of the systems, such as safety and liveness; however, the formal verification is not supported by AADL. Model transformation is commonly applied to provide formal semantics; hence, the AADL model can be verified by a language that supports formal verification in addition to the ability to cover all AADL model behavior. Event-B, with its proof obligation, is increasingly used to model and verify safety-critical systems. This paper presents the transformation of the AADL model into Event-B, which captures most AADL components and behavioral actions to be effective in the verification of current real-time systems models. Then, we define theorems and invariants of safety and liveness properties to be proven by using the RODIN platform. To demonstrate the efficacy of our method, we model the AADL of movement authority (MA) control of the Chinese Train Control System, transform the AADL model to Event-B and verify its crucial properties.

Virtual timeline: a formal abstraction for verifying preemptive schedulers with temporal isolation

The reliability and security of safety-critical real-time systems are of utmost importance because the failure of these systems could incur severe consequences (e.g., loss of lives or failure of a mission). Such properties require strong isolation between components and they rely on enforcement mechanisms provided by the underlying operating system (OS) kernel. In addition to spatial isolation which is commonly provided by OS kernels to various extents, it also requires temporal isolation, that is, properties on the schedule of one component (e.g., schedulability) are independent of behaviors of other components. The strict isolation between components relies critically on algorithmic properties of the concrete implementation of the scheduler, such as timely provision of time slots, obliviousness to preemption, etc. However, existing work either only reasons about an abstract model of the scheduler, or proves properties of the scheduler implementation that are not rich enough to establish the isolation between different components. In this paper, we present a novel compositional framework for reasoning about algorithmic properties of the concrete implementation of preemptive schedulers. In particular, we use virtual timeline , a variant of the supply bound function used in real-time scheduling analysis, to specify and reason about the scheduling of each component in isolation. We show that the properties proved on this abstraction carry down to the generated assembly code of the OS kernel. Using this framework, we successfully verify a real-time OS kernel, which extends mCertiKOS, a single-processor non-preemptive kernel, with user-level preemption, a verified timer interrupt handler, and a verified real-time scheduler. We prove that in the absence of microarchitectural-level timing channels, this new kernel enjoys temporal and spatial isolation on top of the functional correctness guarantee. All the proofs are implemented in the Coq proof assistant.

An energy-efficient time-triggered scheduling algorithm for mixed-criticality systems

Real-time safety-critical systems are getting more complicated due to the introduction of mixed-criticality systems. The increasing use of mixed-criticality systems has motivated the real-time systems research community to investigate various non-functional aspects of these systems. Energy consumption minimization is one such aspect which is just beginning to be explored. In this paper, we propose a time-triggered dynamic voltage and frequency scaling (DVFS) algorithm for uniprocessor mixed-criticality systems. We show that our algorithm outperforms the predominant existing algorithm which uses DVFS for mixed-criticality systems with respect to minimization of energy consumption. In addition, ours is the first energy-efficient time-triggered algorithm for mixed-criticality systems. We prove an optimality result for the proposed algorithm with respect to energy consumption. Then we extend our algorithm for tasks with dependency constraints.

Resource-Aware Scheduling for Dependable Multicore Real-Time Systems: Utilization Bound and Partitioning Algorithm

As the computing devices and software executions are susceptible to manifold faults, fault tolerance has been an important research topic in safety-critical real-time systems. Moreover, multicore processors have recently emerged as prevailing computing engines for modern embedded systems. However, there exists rather rare work on the fault-tolerant scheduling of real-time tasks executing on multicores with shared resources, where the task synchronization originated from resource access contention may significantly degrade the schedulability of task system. With the focus on the partitioned-EDF scheduler with the MSRP (Multiprocessor Stack Resource Policy) protocol and primary/backup recovery mechanism, we first investigate a utilization bound and then identify its anomaly where the bound may decrease when more cores are deployed. Next, following the insights gained by the analysis of the bound, we propose a reliability and synchronization aware task partitioning algorithm (RSA-TPA) together with an efficient version to implement the joint management of task synchronization and system reliability, where several resource-oriented heuristics are developed to improve both the schedulability performance and workload balancing. The extensive simulation results show that the RSA-TPA schemes can obtain higher acceptance ratio (e.g., 60 percent more) and generate more balanced partitions, when compared to the existing schemes that consider either reliability management or task synchronization. Finally, with the different fault arrival rates being considered, the actual implementation in Linux kernel further demonstrates the applicability of RSA-TPA that has lower run-time overhead (e.g., 20 percent less) in comparison with other mapping algorithms.

Fault-tolerant real-time tasks scheduling with dynamic fault handling

Abstract Predictable performance when coping with transient failures is of paramount importance in safety-critical real-time systems. Various software fault-tolerant techniques are employed towards this goal among which check-pointing is a relatively cost-effective scheme. In this paper, we propose an efficient fault-tolerant scheduling framework with run-time fault handling protocol, where criticality levels can be adaptively inserted for fault handling according to run-time fault workload. In contrast to prior works which apply with task re-execution strategy, the proposed framework adaptively determines on-demand re-executions only on the faulty checkpoint segments, rather than on the whole job. Towards this, a unified overrun handling protocol is developed to handle fault recovery adaptively to avoid over-provisioning of resources. In addition, we develop an off-line schedulability analysis technique for the proposed scheduling algorithm. The simulation results show that our fault-tolerant scheduling framework can bring up to 81% improvement in supporting low-criticality service without sacrifice in the MC-schedulability compared with the existing techniques.

Reduced latency DRAM for multi-core safety-critical real-time systems

Predictable execution time upon accessing shared memories in multi-core real-time systems is a stringent requirement. A plethora of existing works focus on the analysis of Double Data Rate Dynamic Random Access Memories (DDR DRAMs), or redesigning its memory to provide predictable memory behavior. In this paper, we show that DDR DRAMs by construction suffer inherent limitations associated with achieving such predictability. These limitations lead to (1) highly variable access latencies that fluctuate based on various factors such as access patterns and memory state from previous accesses, and (2) overly pessimistic latency bounds. As a result, DDR DRAMs can be ill-suited for some real-time systems that mandate a strict predictable performance with tight timing constraints. Targeting these systems, we promote an alternative off-chip memory solution that is based on the emerging Reduced Latency DRAM (RLDRAM) protocol, and propose a predictable memory controller (RLDC) managing accesses to this memory. Comparing with the state-of-the-art predictable DDR controllers, the proposed solution provides up to $$\mathbf{11 }\times $$ less timing variability and $$\mathbf{6.4 }\times $$ reduction in the worst case memory latency.

A formal approach to AADL model-based software engineering

Formal methods have become a recommended practice in safety-critical software engineering. To be formally verified, a system should be specified with a specific formalism such as Petri nets, automata and process algebras, which requires a formal expertise and may become complex especially with large systems. In this paper, we report our experience in the formal verification of safety-critical real-time systems. We propose a formal mapping for a real-time task model using the LNT language, and we describe how it is used for the integration of a formal verification phase in an AADL model-based development process. We focus on real-time systems with event-driven tasks, asynchronous communication and preemptive fixed-priority scheduling. We provide a complete tool-chain for the automatic model transformation and formal verification of AADL models. Experimentation illustrates our results with the Flight control system and Line follower robot case studies.