Runtime Monitoring Research Articles

Model-based Testing and Monitoring using AgileUML

Abstract In this paper we present an approach for the model-based testing (MBT) and runtime monitoring of applications using the UML-RSDS language and AgileUML toolset. This approach uses class invariants and operation pre/post specifications to guide the choice of test cases, and synthesises monitoring code from temporal logic specifications of application behaviour. We give examples of the use of the techniques for mobile apps and financial systems.

Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

As deep learning continues to dominate all state-of-the-art computer vision tasks, it is increasingly becoming an essential building block for robotic perception. This raises important questions concerning the safety and reliability of learning-based perception systems. There is an established field that studies safety certification and convergence guarantees of complex software systems at design-time. However, the unknown future deployment environments of an autonomous system and the complexity of learning-based perception make the generalization of design-time verification to run-time problematic. In the face of this challenge, more attention is starting to focus on run-time monitoring of performance and reliability of perception systems with several trends emerging in the literature in the face of this challenge. This paper attempts to identify these trends and summarize the various approaches to the topic.

Out-of-Distribution Detection for Deep Neural Networks With Isolation Forest and Local Outlier Factor

Deep Neural Networks (DNNs) are extensively deployed in today’s safety-critical autonomous systems thanks to their excellent performance. However, they are known to make mistakes unpredictably, e.g., a DNN may misclassify an object if it is used for perception, or issue unsafe control commands if it is used for planning and control. One common cause for such unpredictable mistakes is Out-of-Distribution (OOD) input samples, i.e., samples that fall outside of the distribution of the training dataset. We present a framework for OOD detection based on outlier detection in one or more hidden layers of a DNN with a runtime monitor based on either Isolation Forest (IF) or Local Outlier Factor (LOF). Performance evaluation indicates that LOF is a promising method in terms of both the Machine Learning metrics of precision, recall, F1 score and accuracy, as well as computational efficiency during testing.

Security Verification for Cyber-Physical Systems Using Model Checking

A malicious attack may endanger human life or pollute environment on a cyber-physical system (CPS). However, successfully attacking a CPS needs not only the knowledge of information technology (IT) but also the domain knowledge of the system’s operation technology (OT). Therefore, it is critical to identify the vulnerabilities of a CPS. This paper proposes a systematic method for the security verification of a CPS, focusing on OT by using model checking with UPPAAL, so as to enhance cyber security. In our security analysis, we considered unsafe situations to be the result of a potentially effective security attack. Thus, we suggested a systematic method to generate security constraints based on the safety constraints (or safety checks) of the CPS and then enhance these security constraints by security verification using model checking with UPPAAL. UPPAAL’s simulation tool can perform a detailed search for each state in various possible model combinations and can explore human-computer interactions more deeply. The contributions of our method are as follows: First, a systematic method is proposed to generate security constraints based on the overall safety requirements at the OT level. Second, the security constraints thus generated can be used for run-time monitoring to identify the possible security attacks when they are violated. Third, this paper proposes to augment normal system modeling with a suggested Attack Module to simulate the potential OT attacks. Finally, the verification results may be used in the following twofold directions: to identify the vulnerabilities for possible design improvements and to suggest the further additions of security constraints.

2.5D Root of Trust: Secure System-Level Integration of Untrusted Chiplets

Dedicated, after acceptance and publication, in memory of the late Vassos Soteriou. For the first time, we leverage the 2.5D interposer technology to establish system-level security in the face of hardware- and software-centric adversaries. More specifically, we integrate chiplets (i.e., third-party hard intellectual property of complex functionality, like microprocessors) using a security-enforcing interposer. Such hardware organization provides a robust 2.5D root of trust for trustworthy, yet powerful and flexible, computation systems. The security paradigms for our scheme, employed firmly by design and construction, are: 1) stringent physical separation of trusted from untrusted components, and 2) runtime monitoring. The system-level activities of all untrusted commodity chiplets are checked continuously against security policies via physically separated security features. Aside from the security promises, the good economics of outsourced supply chains are still maintained; the system vendor is free to procure chiplets from the open market, while only producing the interposer and assembling the 2.5D system oneself. We showcase our scheme using the Cortex-M0 core and the AHB-Lite bus by ARM, building a secure 64-core system with shared memories. We evaluate our scheme through hardware simulation, considering different threat scenarios. Finally, we devise a physical-design flow for 2.5D systems, based on commercial-grade design tools, to demonstrate and evaluate our 2.5D root of trust.

Towards Real-time, On-board, Hardware-supported Sensor and Software Health Management for Unmanned Aerial Systems

For unmanned aerial systems (UAS) to be successfully deployed and integrated within the national airspace, it is imperative that they possess the capability to effectively complete their missions without compromising the safety of other aircraft, as well as persons and property on the ground. This necessity creates a natural requirement for UAS that can respondto uncertain environmental conditions and emergent failures in real-time, with robustness and resilience close enough to those of manned systems. We introduce a system that meets this requirement with the design of a real-time onboard system health management (SHM) capability to continuously monitor sensors, software, and hardware components. This system can detect and diagnose failures and violations of safety or performance rules during the flight of a UAS. Our approach to SHM is three-pronged, providing: (1) real-time monitoring of sensor and software signals; (2) signal analysis, preprocessing, and advanced on-the-fly temporal and Bayesian probabilistic fault diagnosis; and (3) an unobtrusive, lightweight, read-only, low-power realization using Field Programmable Gate Arrays (FPGAs) that avoids overburdening limited computing resources or costly re-certification of flight software. We call this approach rt-R2U2, a name derived from its requirements. Our implementation provides a novel approach of combining modular building blocks, integrating responsive runtime monitoring of temporal logic system safety requirements with model-based diagnosis and Bayesian network-based probabilistic analysis. We demonstrate this approach using actual flight data from theNASA Swift UAS.

A Dual-Core RISC-V Vector Processor With On-Chip Fine-Grain Power Management in 28-nm FD-SOI

This work demonstrates a dual-core RISC-V system-on-chip (SoC) with integrated fine-grain power management. The 28-nm fully depleted silicon-on-insulator (FD-SOI) SoC integrates switched-capacitor voltage converters and 4-Gb/s off-chip serial links. The SoC runs applications with operating system support on dual RISC-V Rocket cores with vector accelerators. Runtime monitoring of microarchitectural counters allows prediction of future compute intensity, enabling the voltage state of the managed core to be adjusted quickly to optimize energy efficiency without sacrificing overall performance.

Compositional Probabilistic Analysis of Temporal Properties Over Stochastic Detectors

Runtime monitoring is a vital part of safety-critical systems. However, early stage assurance of monitoring quality is currently limited: it relies either on complex models that might be inaccurate in unknown ways or on data that would only be available once the system has been built. To address this issue, we propose a compositional framework for modeling and analysis of noisy monitoring systems. Our novel 3-value detector model uses probability spaces to represent atomic (noncomposite) detectors, and it composes them into a temporal logic-based monitor. The error rates of these monitors are estimated by our analysis engine, which combines symbolic probability algebra, independence inference, and estimation from labeled detection data. Our evaluation on an autonomous underwater vehicle found that our framework produces accurate estimates of error rates while using only detector traces, without any monitor traces. Furthermore, when data are scarce, our approach shows higher accuracy than noncompositional data-driven estimates from monitor traces. Thus, this article enables accurate evaluation of logical monitors in early design stages before deploying them.

Specification-Guided Automated Debugging of CPS Models

Simulink/Stateflow is the de facto tool for developing software for safety-critical real-time cyber-physical systems (CPSs). In Simulink, the model of a CPS is captured in a block diagram-based language, the model is simulated using the associated simulators and then the software code is generated automatically for the embedded controller. The presence of a bug in the Simulink model may lead to catastrophic failure during the execution of the system developed based on the model. Unlike in application software, finding bugs in Simulink models is challenging due to the hybrid nature of the model. We present an automated debugging methodology of a CPS model captured in Simulink. Our methodology has two main components-bug localization and model repair. For bug localization, we capture the requirements of the system in signal temporal logic (STL) and employ the runtime monitoring technique to generate a trace that violates the specification. The violating trace is used to identify the internal signals that have the potential to contribute to the violation. For precise bug localization by narrowing down the offending signals, we employ a matrix decomposition technique to find the signals contributing to the bug accurately. Our bug localization technique is precise enough to enable us to repair the model. If the bug is due to an inappropriate value for a model parameter, we employ a parameter tuning method to find a value for the parameter that repairs the model automatically. We carry out numerous case studies on Simulink models obtained from different sources and demonstrate that our automated debugging technology can fix the bugs in the Simulink models effectively.

Conventional and machine learning approaches as countermeasures against hardware trojan attacks

Every year, the rate at which technology is applied on areas of our everyday life is increasing at a steady pace. This rapid development drives the technology companies to design and fabricate their integrated circuits (ICs) in non-trustworthy outsourcing foundries to reduce the cost, thus, leaving space for a synchronous form of virus, known as Hardware Trojan (HT), to be developed. HTs leak encrypted information, degrade device performance or lead to total destruction. To reduce the risks associated with these viruses, various approaches have been developed aiming to prevent and detect them, based on conventional or machine learning methods. Ideally, any undesired modification made to an IC should be detectable by pre-silicon verification/simulation and post-silicon testing. The infected circuit can be inserted in different stages of the manufacturing process, rendering the detection of HTs a complicated procedure. In this paper, we present a comprehensive review of research dedicated to countermeasures against HTs embedded into ICs. The literature is grouped in four main categories; (a) conventional HT detection approaches, (b) machine learning for HT countermeasures, (c) design for security and (d) runtime monitor.

Multi-valued Verification of Strategic Ability

Some multi-agent scenarios call for the possibility of evaluating specifications in a richer domain of truth values. Examples include runtime monitoring of a temporal property over a growing prefix of an infinite path, inconsistency analysis in distributed databases, and verification methods that use incomplete anytime algorithms, such as bounded model checking. In this paper, we present multi-valued alternating-time temporal logic ( mv-ATL → ∗ ), an expressive logic to specify strategic abilities in multi-agent systems. It is well known that, for branchingtime logics, a general method for model-independent translation from multi-valued to two-valued model checking exists. We show that the method cannot be directly extended to mv-ATL → ∗ . We also propose two ways of overcoming the problem. Firstly, we identify constraints on formulas for which the model-independent translation can be suitably adapted. Secondly, we present a model-dependent reduction that can be applied to all formulas of mv-ATL → ∗ . We show that, in all cases, the complexity of verification increases only linearly when new truth values are added to the evaluation domain. We also consider several examples that show possible applications of mv-ATL → ∗ and motivate its use for model checking multi-agent systems.

CPU Elasticity to Mitigate Cross-VM Runtime Monitoring

In this paper, We present a new technique that offers lightweight, general, and elastic protection against Crum (Cross-VM runtime monitoring) attacks. Our protection, called Crease (CPU Resource Elasticity as a Service), enables a VM (called principal ) to purchase a higher clock rate from the cloud, through lowering the frequency of a malicious VM (called peer ), to support its security-critical operations within a short period. During that period, the weakened peer becomes unable to catch up with the pace of the strengthened principal, therefore losing the capability to effectively collect its sensitive information. In the meantime, our approach can also make up for the performance impact on the peer through refunding schedule credits or service credits, in line with the service level agreement of today's cloud. At the center of our design is the novel application of on-demand frequency scaling and schedule quantum randomization, together with a situation-awareness mechanism that dynamically assesses the security risk posed by the peer. We analyzed the security guarantee of our design, implemented a prototype and evaluated it on a well-known Crum attack (an LLC side-channel attack) and various workloads. Our study shows that Crease is effective at protecting the principal, with only a small impact on the peer's operations.

Can determinism and compositionality coexist in RML?

Runtime verification (RV) consists in dynamically verifying that the event traces generated by single runs of a system under scrutiny (SUS) are compliant with the formal specification of its expected properties. RML (Runtime Monitoring Language) is a simple but expressive Domain Specific Language for RV; its semantics is based on a trace calculus formalized by a deterministic rewriting system which drives the implementation of the interpreter of the monitors generated by the RML compiler from the specifications. While determinism of the trace calculus ensures better performances of the generated monitors, it makes the semantics of its operators less intuitive. In this paper we move a first step towards a compositional semantics of the RML trace calculus, by interpreting its basic operators as operations on sets of instantiated event traces and by proving that such an interpretation is equivalent to the operational semantics of the calculus.

A systematic map on verification and validation of emergent behavior in software engineering research

Context:Emergent behavior cannot be attributed to one individual system alone but arises in the interplay of various systems, components etc. Ensuring the correctness of emergent behavior is a well-known challenge that has been addressed by research in various subfields of software engineering. Objective:This paper aims at providing a unified view on the research activities conducted and research contributions made on verification and validation of emergent behavior. Methods:We have conducted a systematic mapping study on the topic of verification and validation of emergent behavior. We applied a combined search strategy using manual, database, and snowball search. In total we investigated 7211 papers, from these 168 relevant papers have been included and classified. Results:Results show an increasing interest in the topic of verification and validation of emergent behavior. As only little validation and evaluation research has been conducted, the field can be considered still immature. There exist different verification and validation techniques used in the various solution approaches such as model checking, simulation, or runtime monitoring. It stands out that even though research is published in different software engineering fields and subfields no verification or validation technique can be attributed solely to a single field.

Evolution in dynamic software product lines

AbstractMany software systems today provide support for adaptation and reconfiguration at runtime, in response to changes in their environment. Such adaptive systems are designed to run continuously and may not be shut down for reconfiguration or maintenance tasks. The variability of such systems has to be explicitly managed, together with mechanisms that control their runtime adaptation and reconfiguration. Dynamic software product lines (DSPLs) can help to achieve this. However, dealing with evolution is particularly challenging in a DSPL, as changes made at runtime can easily lead to inconsistencies. This paper describes the challenges of evolving DSPLs using an example cyber‐physical system for home automation. We discuss the shortcomings of existing work and present a reference architecture to support DSPL evolution. To demonstrate its feasibility and flexibility, we implemented the proposed reference architecture for two different DSPLs: the aforementioned cyber‐physical system, which uses feature models to describe its variability, and a runtime monitoring infrastructure, which is based on decision models. To assess the industrial applicability of our approach, we also implemented the reference architecture for a real‐world DSPL, an automation software system for injection molding machines. Our results provide evidence on the flexibility, performance, and industrial applicability of our approach.

Assured Runtime Monitoring and Planning: Toward Verification of Neural Networks for Safe Autonomous Operations

Autonomous systems operating in uncertain envi ronments under the effects of disturbances and noises can reach unsafe states even while using finetuned controllers and precise sensors and actuators. To provide safety guarantees on such systems during motion planning operations, reachability analysis (RA) has been demonstrated to be a powerful tool. RA, however, suffers from computational complexity, especially when dealing with intricate systems characterized by high-order dynamics, making it hard to deploy for runtime monitoring.

Geofencing requirements for onboard safe operation monitoring

The new concept for operation of drones, published by EASA in 2015, enables new ways to influence and possibly reduce the necessary safety targets of certain system components without reducing the overall safety of the unmanned aircraft system (UAS). Based on the safety assessment, the specific category enables new aircraft system architectures and mission designs. In this context, this paper analyzes runtime monitoring as a strategy to contain the UAS in its operational volume. To assure predefined properties in flight and thus assure the safety of the operation in progress with a high robustness, a formal methodology for safe operation monitoring is utilized. With this approach, this work targets to link the concept of safe operation monitoring with the upcoming regulations regarding the specific category and the specific operation risk assessment (SORA). One particular aspect of this safe operation monitoring is geofencing, the capability to contain a UAS in a previously restricted area. In the regulatory framework of a specific operation, risk assessment is required and so is the containment of the UAS in its operational volume. The functional and safety requirements for geofencing regarding their impact on the underlying specific operation risk assessment are discussed. To facilitate this discussion, a taxonomy of geofencing characteristics is derived based on a literature survey. Consequently, the geofencing requirements are assessed regarding their robustness and applicability for certification purposes. As a result, by monitoring the integrity of the system at runtime using geofencing as an example, it is investigated if the requirements and thus costs of development and certification process for the remaining components can be reduced.

Safety Assurance Concepts for Automated Driving Systems

<div class="section abstract"><div class="htmlview paragraph">Automated driving systems (ADSs) for road vehicles are being developed that can perform the entire dynamic driving task without a human driver in the loop. However, current regulatory frameworks for assuring vehicle safety may restrict the deployment of ADSs that can use machine learning to modify their functionality while in service. A review was undertaken to identify and assess key initiatives and research relevant to the safety assurance of adaptive safety-critical systems that use machine learning, and to highlight assurance concepts that could benefit from further research. The primary objective was to produce findings and recommendations that can inform policy and regulatory reform relating to ADS safety assurance. Due to the almost infinite number and combination of scenarios that an ADS could encounter, the review found much support for concepts that involve the use of simulation data as virtual evidence of safety compliance, with suggestions of a need to assure simulation tools and models. Real-world behavioural competency testing was also commonly proposed, although noting this concept has its limitations. The concept of whole-of-life assurance was identified, supported by various versions of dynamic runtime monitoring, verification and validation. Concerns regarding Artificial Intelligence (AI) robustness were raised, particularly regarding adversarial inputs and unmodelled scenarios that are essentially unknown unknowns. Further, the concept of explainable AI was highlighted as having potential to provide evidence from an ADS that could support safety assurance and regulatory compliance. While each of the identified assurance concepts should be considered when developing ADS safety assurance framework options, it is recommended that further research on each concept should be progressed.</div></div>

Trustworthy isolation of DMA devices

We present a mechanism to trustworthy isolate I/O devices with direct memory access (DMA), which ensures that an isolated I/O device cannot access sensitive memory regions. As a demonstrating platform, we use the network interface controller (NIC) of an embedded system. We develop a run-time monitor that forces NIC reconfigurations, defined by untrusted software, to satisfy a security rule. We formalized the NIC in the HOL4 interactive theorem prover and we verified the design of the isolation mechanism. The verification is based on an invariant that is proved to be preserved by all NIC operations and that ensures that all memory accesses address allowed memory regions only. We demonstrate our approach by extending an existing Virtual Machine Introspection (VMI) with the monitor. The resulting platform prevents code injection in a connected and untrusted Linux.

Consistency analysis and flow secure enforcement of SELinux policies

SELinux policies used in practice contain tens of thousands of rules, making it hard to comprehend their impact on the security and to verify whether they actually meet the intended security goals. In this paper, we describe an approach for reasoning about the consistency of a given SELinux policy by analyzing the information flows caused by it. For this purpose, we model SELinux policy rules using the Readers-Writers Flow Model (RWFM). We have used this approach to implement a static policy analysis tool as well as a run-time monitor. The static policy analysis tool identifies all the possible indirect flows in a given policy and then filters out those indirect flows that pose a high threat. Given an indirect flow, the tool can also identify the sequences of accesses that cause the indirect flow. The tool also ranks the rules and domains based on the number of policy violations they cause. Thus, the static analysis tool is useful for policy writers to develop flow secure policies. The run-time monitor, on the other hand, keeps track of the information flows in an SELinux system and detects indirect flows dynamically. This helps in ensuring flow secure enforcement of a given SELinux policy as per the specification. The efficiency and efficacy of our implementations are demonstrated through experimental analysis on large, real-life policies.