Runtime Monitoring Research Articles

Runtime Monitoring of ML-Based Scheduling Algorithms Toward Robust Domain-Specific SoCs

Runtime Monitoring of ML-Based Scheduling Algorithms Toward Robust Domain-Specific SoCs

On Optimizing Resources for Real‐Time End‐to‐End Machine Learning in Heterogeneous Edges

ABSTRACTDeploying end‐to‐end ML applications on edge resources becomes a viable solution to achieve performance and data regulations. With the microservice architecture, these applications can scale dynamically, improving service availability under dynamic workloads. However, orchestrating multiple end‐to‐end ML applications within heterogeneous edge environments must deal with numerous challenges while sharing computing resources. Prevalent orchestration tools/frameworks supporting edge ML serving are inefficient in provisioning methods due to constrained resources, diverse resource demands and utilization patterns. In this work, we present a provisioning method to optimize resource utilization for end‐to‐end ML applications on a heterogeneous edge. By profiling all microservices within the application, we estimate scales and allocate them on desired hardware platforms with sufficient resources when considering their runtime utilization patterns. We also provide several practical analyses on runtime monitoring metrics to detect and mitigate resource contentions, guaranteeing performance. The experiments with three real‐world ML applications demonstrate the practicality of our method on a heterogeneous edge cluster of Raspberry Pis and Jetson Developer Kits.

Statistical Analysis and Runtime Monitoring for an AI-based Autonomous Centerline Tracking System

Autonomous Centerline Tracking (ACT) enables an unmanned aircraft to be guided down the center of the runway, using a camera-based Deep Neural Network (DNN). ACT is safety-critical. The EASA Guidelines for machine-learning based systems list numerous assurance objectives that must be met toward certification and V&V. We extend our statistical analysis framework SYSAI to support meeting assurance objectives for a complex safety-critical system with AI components, and describe a combination with a runtime monitoring architecture that also supports advanced risk mitigation to support safety assurance of a complex AI-based aerospace system.

Vamos: Middleware for best-effort third-party monitoring

As the complexity and criticality of software increase every year, so does the importance of runtime monitoring. Third-party and best-effort monitoring are especially valuable, yet under-explored areas of runtime monitoring. In this context, third-party monitoring means monitoring with a limited knowledge of the monitored software (as it has been developed by a third party). Best-effort monitoring keeps pace with the monitored software at the cost of possibly imprecise verdicts when keeping up with the monitored software would not be feasible. Most existing monitoring frameworks do not support the combination of third-party and best-effort monitoring because they either require the full access to the monitored code or the ability to process all observable events, or both.We present a middleware framework, Vamos, for the runtime monitoring of software. Vamos is explicitly designed to support third-party and best-effort scenarios. The design goals of Vamos are (i) efficiency (tracing events with low overhead), (ii) flexibility (the ability to monitor a variety of different event channels, and to connect to a wide range of monitors), and (iii) ease-of-use. To achieve its goals, Vamos combines aspects of event broker and event recognition systems with aspects of stream processing systems.We implemented a prototype toolchain for Vamos and conducted a set of experiments demonstrating the usability of the scheme. The results indicate that Vamos enables writing useful yet efficient monitors, and simplifies key aspects of setting up a monitoring system from scratch.

Formal Methods in Industry

Formal methods encompass a wide choice of techniques and tools for the specification, development, analysis, and verification of software and hardware systems. Formal methods are widely applied in industry, in activities ranging from the elicitation of requirements and the early design phases all the way to the deployment, configuration, and runtime monitoring of actual systems. Formal methods allow one to precisely specify the environment in which a system operates, the requirements and properties that the system should satisfy, the models of the system used during the various design steps, and the code embedded in the final implementation, as well as to express conformance relations between these specifications. We present a broad scope of successful applications of formal methods in industry, not limited to the well-known success stories from the safety-critical domain, like railways and other transportation systems, but also covering other areas such as lithography manufacturing and cloud security in e-commerce, to name but a few. We also report testimonies from a number of representatives from industry who, either directly or indirectly, use or have used formal methods in their industrial project endeavours. These persons are spread geographically, including Europe, Asia, North and South America, and the involved projects witness the large coverage of applications of formal methods, not limited to the safety-critical domain. We thus make a case for the importance of formal methods, and in particular of the capacity to abstract and mathematical reasoning that are taught as part of any formal methods course. These are fundamental Computer Science skills that graduates should profit from when working as computer scientists in industry, as confirmed by industry representatives.

Timing Side-Channel Mitigation via Automated Program Repair

Side-channel vulnerability detection has gained prominence recently due to Spectre and Meltdown attacks. Techniques for side-channel detection range from fuzz testing to program analysis and program composition. Existing side-channel mitigation techniques repair the vulnerability at the IR/binary level or use runtime monitoring solutions. In both cases, the source code itself is not modified, can evolve while keeping the vulnerability, and the developer would get no feedback on how to develop secure applications in the first place. Thus, these solutions do not help the developer understand the side-channel risks in her code and do not provide guidance to avoid code patterns with side-channel risks. In this paper, we present Pendulum , the first approach for automatically locating and repairing side-channel vulnerabilities in the source code, specifically for timing side channels. Our approach uses a quantitative estimation of found vulnerabilities to guide the fix localization, which goes hand-in-hand with a pattern-guided repair. Our evaluation shows that Pendulum can repair a large number of side-channel vulnerabilities in real-world applications. Overall, our approach integrates vulnerability detection, quantization, localization, and repair into one unified process. This also enhances the possibility of our side-channel mitigation approach being adopted into programming environments.

MarMot: Metamorphic Runtime Monitoring of Autonomous Driving Systems

Autonomous Driving Systems (ADSs) are complex Cyber-Physical Systems (CPSs) that must ensure safety even in uncertain conditions. Modern ADSs often employ Deep Neural Networks (DNNs), which may not produce correct results in every possible driving scenario. Thus, an approach to estimate the confidence of an ADS at runtime is necessary to prevent potentially dangerous situations. In this paper we propose MarMot, an online monitoring approach for ADSs based on Metamorphic Relations (MRs), which are properties of a system that hold among multiple inputs and the corresponding outputs. Using domain-specific MRs, MarMot estimates the uncertainty of the ADS at runtime, allowing the identification of anomalous situations that are likely to cause a faulty behavior of the ADS, such as driving off the road. We perform an empirical assessment of MarMot with five different MRs, using two different subject ADSs, including a small-scale physical ADS and a simulated ADS. Our evaluation encompasses the identification of both external anomalies, e.g., fog, as well as internal anomalies, e.g., faulty DNNs due to mislabeled training data. Our results show that MarMot can identify up to 65% of the external anomalies and 100% of the internal anomalies in the physical ADS, and up to 54% of the external anomalies and 88% of the internal anomalies in the simulated ADS. With these results, MarMot outperforms or is comparable to other state-of-the-art approaches, including SelfOracle, Ensemble, and MC Dropout-based ADS monitors.

A survey of safety and trustworthiness of large language models through the lens of verification and validation

Large language models (LLMs) have exploded a new heatwave of AI for their ability to engage end-users in human-level conversations with detailed and articulate answers across many knowledge domains. In response to their fast adoption in many industrial applications, this survey concerns their safety and trustworthiness. First, we review known vulnerabilities and limitations of the LLMs, categorising them into inherent issues, attacks, and unintended bugs. Then, we consider if and how the Verification and Validation (V&V) techniques, which have been widely developed for traditional software and deep learning models such as convolutional neural networks as independent processes to check the alignment of their implementations against the specifications, can be integrated and further extended throughout the lifecycle of the LLMs to provide rigorous analysis to the safety and trustworthiness of LLMs and their applications. Specifically, we consider four complementary techniques: falsification and evaluation, verification, runtime monitoring, and regulations and ethical use. In total, 370+ references are considered to support the quick understanding of the safety and trustworthiness issues from the perspective of V&V. While intensive research has been conducted to identify the safety and trustworthiness issues, rigorous yet practical methods are called for to ensure the alignment of LLMs with safety and trustworthiness requirements.

Computing Unsatisfiable Cores for LTLf Specifications

Linear-time temporal logic on finite traces (LTLf) is rapidly becoming a de-facto standard to produce specifications in many application domains (including planning, business process management, run-time monitoring, and reactive synthesis). Several studies have challenged the satisfiability problem thus far. In this paper, we focus instead on unsatisfiable LTLf specifications, with the objective of extracting the subset of formulae that cause inconsistencies within them, i.e., the unsatisfiable cores. We provide four algorithms to this end, which leverage the adaptation of a range of state-of-the-art algorithms to LTLf satisfiability checking. We implement those algorithms extending the respective implementations and carry out an experimental evaluation on a set of reference benchmarks, restricting to the unsatisfiable specifications. The results put in evidence that the different algorithms and tools exhibit complementary features determining their efficiency and efficacy. Indeed, our findings suggest exploring different strategies and algorithmic solutions for the extraction of unsatisfiable cores from LTLf specifications, thus confirming the challenging and multi-faceted nature of this problem.

ALERT: A lightweight defense mechanism for enhancing DNN robustness against T-BFA

DNNs have become pervasive in many security–critical scenarios such as autonomous vehicles and medical diagnoses. Recent studies reveal the susceptibility of DNNs to various adversarial attacks, among which weight Bit-Flip Attacks (BFA) is emerging as a significant security concern. Moreover, Targeted Bit-Flip Attacks (T-BFA), as a novel variant of BFA, can stealthily alter specific source–target classifications while preserving accurate classifications of non-target classes, posing a more severe threat. However, due to the inadequate consideration for T-BFA’s “targeted” characteristic, existing defense mechanisms tend to perform over-protection/-modification to the network, leading to significant defense overheads or non-negligible DNN accuracy reduction.In this work, we propose ALERT, ALightweight defense mechanism for Enhancing DNN Robustness against T-BFA while maintaining network accuracy. Firstly, fully understanding the key factors that dominate the misclassification among source–target class pairs, we propose a Source-Target-Aware Searching (STAS) method to accurately identify the vulnerable weights under T-BFA. Secondly, leveraging the intrinsic redundancy characteristic of DNNs, we propose a weight random switch mechanism to reduce the exposure of vulnerable weights, thereby weakening the expected impact of T-BFA. Striking a delicate balance between enhancing robustness and preserving network accuracy, we develop a metric to meticulously select candidate weights. Finally, to further enhance the DNN robustness, we present a lightweight runtime monitoring mechanism for detecting T-BFA through weight signature verification, and dynamically optimize the weight random switch strategy accordingly. Evaluation results demonstrate that our proposed method effectively enhances the robustness of DNNs against T-BFA while maintaining network accuracy. Compared with the baseline, our method can tolerate 6.7× more flipped bits with negligible accuracy loss (<0.1% in ResNet-50).

DHRCA: A Design of Security Architecture Based on Dynamic Heterogeneous Redundant for System on Wafer

System on Wafer (SoW) based on chiplets may be implanted with hardware Trojans (HTs) by untrustworthy third-party chiplet vendors. However, traditional HTs protection techniques cannot guarantee complete protection against HTs, which poses a great challenge to the hardware security of SoW. In this paper, we propose a computing architecture based on endogenous security theory—dynamic heterogeneous redundant computing architecture (DHRCA) that can tolerate and detect HTs at runtime. The security of our approach is analyzed by building a generalized stochastic coloring petri net (GSCPN) model of DHRCA. The simulation results based on the GSCPN model show that our method can improve the system security probability to 0.8690 and the system availability probability to 0.9750 in the steady state compared with typical triple-mode redundancy and runtime monitoring methods. Furthermore, the impact of different attack and defense strategies on system security of different methods is simulated and analyzed in this paper.

ZoneTrace: A Zone Monitoring Tool for F2FS on ZNS SSDs

We present ZoneTrace , a runtime monitoring tool for the Flash-Friendly File System (F2FS) on Zoned Namespace (ZNS) SSDs. ZNS SSD organizes its storage into zones of sequential write access. Due to ZNS SSD’s sequential write nature, F2FS is a log-structured file system that has recently been adopted to support ZNS SSDs. To present the space management with the zone concept between F2FS and the underlying ZNS SSD, we developed ZoneTrace , a tool that enables users to visualize and analyze the space management of F2FS on ZNS SSDs. ZoneTrace utilizes the extended Berkeley Packet Filter (eBPF) to trace the updated segment bitmap in F2FS and visualize each zone space usage accordingly. Furthermore, ZoneTrace is able to analyze on file fragmentation in F2FS and provides users with informative fragmentation histogram to serve as an indicator of file fragmentation. Using ZoneTrace ’s visualization, we are able to identify the current F2FS space management scheme’s inability to fully optimize space for streaming data recording in autonomous systems, which leads to serious file fragmentation on ZNS SSDs. Our evaluations show that ZoneTrace is lightweight and assists users in getting useful insights for effortless monitoring on F2FS with ZNS SSD with both synthetic and realistic workloads. We believe ZoneTrace can help users analyze F2FS with ease and open up space management research topics with F2FS on ZNS SSDs.

Monitoring of Perception Systems: Deterministic, Probabilistic, and Learning-Based Fault Detection and Identification (Abstract Reprint)

This paper investigates runtime monitoring of perception systems. Perception is a critical component of high-integrity applications of robotics and autonomous systems, such as self-driving cars. In these applications, failure of perception systems may put human life at risk, and a broad adoption of these technologies requires the development of methodologies to guarantee and monitor safe operation. Despite the paramount importance of perception, currently there is no formal approach for system-level perception monitoring. In this paper, we formalize the problem of runtime fault detection and identification in perception systems and present a framework to model diagnostic information using a diagnostic graph. We then provide a set of deterministic, probabilistic, and learning-based algorithms that use diagnostic graphs to perform fault detection and identification. Moreover, we investigate fundamental limits and provide deterministic and probabilistic guarantees on the fault detection and identification results. We conclude the paper with an extensive experimental evaluation, which recreates several realistic failure modes in the LGSVL open-source autonomous driving simulator, and applies the proposed system monitors to a state-of-the-art autonomous driving software stack (Baidu's Apollo Auto). The results show that the proposed system monitors outperform baselines, have the potential of preventing accidents in realistic autonomous driving scenarios, and incur a negligible computational overhead.

An empirical investigation of challenges of specifying training data and runtime monitors for critical software with machine learning and their relation to architectural decisions

The development and operation of critical software that contains machine learning (ML) models requires diligence and established processes. Especially the training data used during the development of ML models have major influences on the later behaviour of the system. Runtime monitors are used to provide guarantees for that behaviour. Runtime monitors for example check that the data at runtime is compatible with the data used to train the model. In a first step towards identifying challenges when specifying requirements for training data and runtime monitors, we conducted and thematically analysed ten interviews with practitioners who develop ML models for critical applications in the automotive industry. We identified 17 themes describing the challenges and classified them in six challenge groups. In a second step, we found interconnection between the challenge themes through an additional semantic analysis of the interviews. We explored how the identified challenge themes and their interconnections can be mapped to different architecture views. This step involved identifying relevant architecture views such as data, context, hardware, AI model, and functional safety views that can address the identified challenges. The article presents a list of the identified underlying challenges, identified relations between the challenges and a mapping to architecture views. The intention of this work is to highlight once more that requirement specifications and system architecture are interlinked, even for AI-specific specification challenges such as specifying requirements for training data and runtime monitoring.

Securing Pacemakers using Runtime Monitors over Physiological Signals

Wearable and implantable medical devices (IMDs) are increasingly deployed to diagnose, monitor, and provide therapy for critical medical conditions. Such medical devices are safety-critical cyber-physical systems (CPSs). These systems support wireless features introducing potential security vulnerabilities. Although these devices undergo rigorous safety certification processes, runtime security attacks are inevitable. Based on published literature, IMDs such as pacemakers and insulin infusion systems can be remotely controlled to inject deadly electric shocks and excess insulin, posing a threat to a patient’s life. While prior works based on formal methods have been proposed to detect potential attack vectors using different forms of static analysis, these have limitations in preventing attacks at runtime. This paper discusses a formal framework for detecting cyber-physical attacks on a pacemaker by monitoring its security policies at runtime. We propose a wearable device that senses the Electrocardiogram (ECG) and Photoplethysmogram (PPG) of the body to detect attacks in a pacemaker. To facilitate the design of this device, we map the security policies of a pacemaker w.r.t ECG and PPG, paving the way for designing formal verification monitors for pacemakers for the first time using multiple physiological signals. The proposed monitoring framework allows the synthesis of parallel monitors from a given set of desired security policies, where all the monitors execute concurrently and generate an alarm to the user in the case of policy violation. Our implementation and the performance evaluation results demonstrate the technical feasibility of designing such a wearable device for attack detection in pacemakers. This device is separate from the pacemaker, ensuring no need for re-certification of pacemakers. Our approach is amenable to the application of security patches when new attack vectors are detected, making the approach ideal for runtime monitoring of medical CPSs.

Predictive Monitoring against Pattern Regular Languages

While current bug detection techniques for concurrent software focus on unearthing low-level issues such as data races or deadlocks, they often fall short of discovering more intricate temporal behaviours that can arise even in the absence of such low-level issues. In this paper, we focus on the problem of dynamically analysing concurrent software against high-level temporal specifications such as LTL. Existing techniques for runtime monitoring against such specifications are primarily designed for sequential software and remain inadequate in the presence of concurrency — violations may be observed only in intricate thread interleavings, requiring many re-runs of the underlying software in conjunction with the analysis. Towards this, we study the problem of predictive runtime monitoring , inspired by the analogous problem of predictive data race detection studied extensively recently. The predictive runtime monitoring question asks, given an execution σ, if it can be soundly reordered to expose violations of a specification. In general, this problem may become easily intractable when either the specifications or the notion of reorderings used is complex. In this paper, we focus on specifications that are given in regular languages. Our notion of reorderings is trace equivalence , where an execution is considered a reordering of another if it can be obtained from the latter by successively commuting adjacent independent actions. We first show that, even in this simplistic setting, the problem of predictive monitoring admits a super-linear lower bound of O ( n α ), where n is the number of events in the execution, and α is a parameter describing the degree of commutativity, and typically corresponds to the number of threads in the execution. As a result, predictive runtime monitoring even in this setting is unlikely to be efficiently solvable, unlike in the non-predictive setting where the problem can be checked using a deterministic finite automaton (and thus, a constant-space streaming linear-time algorithm). Towards this, we identify a sub-class of regular languages, called pattern languages (and their extension generalized pattern languages ). Pattern languages can naturally express specific ordering of some number of (labelled) events, and have been inspired by popular empirical hypotheses underlying many concurrency bug detection approaches such as the “small bug depth” hypothesis. More importantly, we show that for pattern (and generalized pattern) languages, the predictive monitoring problem can be solved using a constant-space streaming linear-time algorithm. We implement and evaluate our algorithm PatternTrack on benchmarks from the literature and show that it is effective in monitoring large-scale applications.

Context-aware environment online monitoring for safety autonomous vehicle systems: an automata-theoretic approach

Intelligent Transport System (ITS) is a typical class of Cyber-Physical Systems (CPS), and due to the special characteristics of such systems, higher requirements are placed on system security. Runtime verification is a lightweight verification technique which is used to improve the security of such systems. However, current runtime verification methods often ignore the effects of the physical environment (e.g., the effects of rain, snow, and other weather changes on road conditions), which results in the inability of the monitor to effectively monitor the system according to the changes in the environment. To address this problem, this paper proposes a method for constructing a runtime monitor with environmental context-awareness capability. First, the physical environment factors affecting the system are formally described and constructed into an environment model, then the system statute is transformed into a Büchi automaton, and then a synthesis algorithm combining the environment model and the Büchi automaton is designed based on the network of automatons, and the corresponding monitor is generated. Finally, the proposed method is applied and verified on simulation and real objects. The experimental results show that the monitors generated based on the method of this paper can effectively monitor unsafe events in different environments, thus improving the safety of intelligent driving systems.

MOTEF: A Testing Framework for Runtime Monitoring Infrastructures

MOTEF: A Testing Framework for Runtime Monitoring Infrastructures

Distributed runtime verification of metric temporal properties

Distributed Systems are often composed of geographically separated components, where the clocks may not be perfectly synchronized. As such verifying the correctness of such system properties are a major challenge and are of utmost importance. In this paper, we describe a centralized runtime monitoring technique for distributed system. First, we propose a generalized runtime verification technique for verifying partially synchronous distributed computations for the metric temporal logic (MTL) by exploiting bounded-skew clock synchronization. Second, we introduce a progression-based formula rewriting scheme for monitoring MTL specifications which employs SMT solving techniques and report experimental results. Third, we also quantify each event according to the possible time of occurrence and calculate the probabilistic guarantee for generating the verification verdict. Lastly, we have implemented the entire procedure and report on extensive synthetic experimental results using UPPAAL, a set of cross-chain transactions implemented on Ethereum and an Industrial Control System of a water treatment plant.

Mobility and Dependence-aware QoS Monitoring in Mobile Edge Computing

Mobile edge computing is a new computing paradigm that performs computing on the edge of a network. It provides services to users by deploying edge servers near mobile devices. Services may be unavailable or do not satisfy the needs of users due to changing edge environments. Quality of service (QoS) is commonly employed as a critical means to indicate qualitative status of services. It is particularly important to monitor QoS of services timely and effectively in the mobile edge environment. However, user mobility and dependencies among QoS values often cause the monitoring results to deviate from the real results in the mobile edge environment. Existing QoS monitoring approaches have not taken into account these problems. To address the problems, this paper proposes ghBSRM-MEC (Gaussian hidden BayeSian Runtime Monitoring for Mobile Edge Computing), a novel mobility and dependence-aware QoS monitoring approach for the mobile edge environment. This approach assumes that the QoS attribute values of edge servers obey Gaussian distribution. It constructs a parent property for each property, thus reducing the dependence between properties. During the training stage, a Gaussian Hidden Bayesian classifier is constructed for each edge server. During the monitoring stage, combining with a KNN algorithm, the classifier is changed dynamically based on user mobility to realize QoS monitoring in the mobile edge environment. The experimental results validate the feasibility, effectiveness, and efficiency of ghBSRM-MEC