Runtime Monitoring Research Articles

Collaborative Cluster Configuration for Distributed Data-Parallel Processing: A Research Overview

Many organizations routinely analyze large datasets using systems for distributed data-parallel processing and clusters of commodity resources. Yet, users need to configure adequate resources for their data processing jobs. This requires significant insights into expected job runtimes and scaling behavior, resource characteristics, input data distributions, and other factors. Unable to estimate performance accurately, users frequently overprovision resources for their jobs, leading to low resource utilization and high costs.In this paper, we present major building blocks towards a collaborative approach for optimization of data processing cluster configurations based on runtime data and performance models. We believe that runtime data can be shared and used for performance models across different execution contexts, significantly reducing the reliance on the recurrence of individual processing jobs or, else, dedicated job profiling. For this, we describe how the similarity of processing jobs and cluster infrastructures can be employed to combine suitable data points from local and global job executions into accurate performance models. Furthermore, we outline approaches to performance prediction via more context-aware and reusable models. Finally, we lay out how metrics from previous executions can be combined with runtime monitoring to effectively re-configure models and clusters dynamically.

An FPGA-based Approach to Evaluate Thermal and Resource Management Strategies of Many-core Processors

The continuous technology scaling of integrated circuits results in increasingly higher power densities and operating temperatures. Hence, modern many-core processors require sophisticated thermal and resource management strategies to mitigate these undesirable side effects. A simulation-based evaluation of these strategies is limited by the accuracy of the underlying processor model and the simulation speed. Therefore, we present, for the first time, an field-programmable gate array (FPGA)-based evaluation approach to test and compare thermal and resource management strategies using the combination of benchmark generation, FPGA-based application-specific integrated circuit (ASIC) emulation, and run-time monitoring. The proposed benchmark generation method enables an evaluation of run-time management strategies for applications with various run-time characteristics. Furthermore, the ASIC emulation platform features a novel distributed temperature emulator design, whose overhead scales linearly with the number of integrated cores, and a novel dynamic voltage frequency scaling emulator design, which precisely models the timing and energy overhead of voltage and frequency transitions. In our evaluations, we demonstrate the proposed approach for a tiled many-core processor with 80 cores on four Virtex-7 FPGAs. Additionally, we present the suitability of the platform to evaluate state-of-the-art run-time management techniques with a case study.

Axiomatizing recursion-free, regular monitors

Monitors are a key tool in the field of runtime verification, where they are used to verify system properties by analyzing execution traces generated by processes. Work on runtime monitoring carried out in a series of papers by Aceto et al. has specified monitors using a variation on the regular fragment of Milner's CCS and studied two trace-based notions of equivalence over monitors, namely verdict and ω-verdict equivalence. This article is devoted to the study of the equational logic of monitors modulo those two notions of equivalence. It presents complete equational axiomatizations of verdict and ω-verdict equivalence for closed and open terms over recursion-free monitors. It is also shown that verdict equivalence has no finite equational axiomatization over open monitors when the set of actions is finite and contains at least two actions.

Distributed runtime verification by past-CTL and the field calculus

Recent trends in the engineering of software-intensive systems increasingly promote the adoption of computation at the edge of the network, in the proximity of where sensing and actuation are performed. Applications are executed directly in IoT devices deployed in the physical environment, possibly with the aid of edge servers: there, interactions are essentially based on physical proximity, and communication with the cloud is sporadic if not absent.The challenge of monitoring the execution of such system, by relying on local interactions only, naturally arises. We address this challenge by proposing a rigorous approach to distributed runtime monitoring for space-based networks of devices. We introduce the past-CTL logic, an extension of past-LTL able to express a variety of properties concerning the knowable past of an event. We formally define a procedure to derive, from a past-CTL formula, monitors that can be distributed on each device and whose collective behaviour verifies the validity of the formula at runtime across space and time. This is achieved by relying on the field calculus, a core programming language used to specify the behaviour of a collection of devices by viewing them as an aggregate computing machine, carrying out altogether a distributed computational process. The field calculus is shown to be a convenient language for our goals, since its functional composition approach provides a natural way of translating in a syntax-directed way properties expressed in a given logic into monitors for such properties. We show that the monitor process executing in each single device runs using local memory, message size, and computation time that are all linear in the size of the formula (1 bit per temporal connective). This matches the efficiency of the best available previous results for (non-distributed) monitors derived from past-LTL formulas. Finally, we empirically evaluate the applicability of the approach to sample problems in distributed computing, through simulated experiments with monitors written through a C++ library implementing the field calculus programming constructs.

Detection of Nonrandom Sign-Based Behavior for Resilient Coordination of Robotic Swarms

Cooperative multirobot systems coordinate their motion by exchanging information through consensus schemes to achieve a common goal. In the event of stealthy cyber attacks, compromised measurements and communication broadcasts can hijack a portion or the entire system toward undesired states. However, in order for these attacks to be effective, they have to exhibit nonrandom characteristics that contradict the expected multirobot system behavior. To deal with these hidden attacks, we propose a runtime monitoring framework that considers the signed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">residual</i> , defined as the difference between the expected and the received information to identify and isolate unexpected nonrandom behavior within the multirobot system. Specifically, the technique that we propose—named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cumulative Sign</i> detector—monitors and compares changes in signed values of residual with their expected occurrences to detect inconsistencies and trigger alarms when an attack is discovered. Our results are validated theoretically by providing detection bounds and are demonstrated with simulations and experiments on swarms of unmanned ground vehicles under different attacks in comparison with state-of-the-art residual-based detection schemes.

SAFETY MANAGER: SERVICIO IOT PARA ASEGURAR EL BUEN COMPORTAMIENTO DE LAS MÁQUINAS Y CONTROLES EN LA FÁBRICA DIGITAL

The digital industry requires increasingly complex and reliable software systems. They need to control and make critical decisions at runtime. As a consequence, the verification and validation of these systems has become a major research challenge. At design and development time, model testing techniques are used while run-time verification aims at verifying that a system satisfies a given property. The latter technique complements the former. The solution presented in this paper targets embedded systems whose software components are designed by state machines defined by Unified Modelling Language (UML). The CRESCO (C++ REflective State-Machines based observable software COmponents) platform generates software components that provide internal information at runtime and the verifier uses this information to check system-level reliability/safety contracts. The verifier detects when a system contract is violated and initiates a safeState process to prevent dangerous scenarios. These contracts are defined by internal information from the software components that make up the system. Thus, as demonstrated in the tested experiment, the robustness of the system is increased. All software components (controllers), such as the verifier, have been deployed as services (producers/consumers) of the Arrowhead IoT platform: the controllers are deployed on local Arrowhead platforms (Edge) and the verifier (Safety Manager) is deployed on an Arrowhead platform (Cloud) that will consume controllers on the Edge and ensure the proper functioning of the plant controllers. Keywords: run-time monitoring, robustness, software components, contracts, software models, state machines

Runtime verification in uncertain environment based on probabilistic model learning.

Runtime verification (RV) is a lightweight approach to detecting temporal errors of system at runtime. It confines the verification on observed trajectory which avoids state explosion problem. To predict the future violation, some work proposed the predictive RV which uses the information from models or static analysis. But for software whose models and codes cannot be obtained, or systems running under uncertain environment, these predictive methods cannot take effect. Meanwhile, RV in general takes multi-valued logic as the specification languages, for example the true, false and inconclusive in three-valued semantics. They cannot give accurate quantitative description of correctness when inconclusive is encountered. We in this paper present a RV method which learns probabilistic model of system and environment from history traces and then generates probabilistic runtime monitor to quantitatively predict the satisfaction of temporal property at each runtime state. In this approach, Hidden Markov Model (HMM) is firstly learned and then transformed to Discrete Time Markov Chain (DTMC). To construct incremental monitor, the monitored LTL property is translated into Deterministic Rabin Automaton (DRA). The final probabilistic monitor is obtained by generating the product of DTMC and DRA, and computing the probabilities for each state. With such a method, one can give early warning once the probability of correctness is lower than a pre-defined threshold, and have the chance to do adjustment in advance. The method has been implemented and experimented on real UAS (Unmanned Aerial Vehicle) simulation platform.

Adaptive Runtime Monitoring of Service Level Agreement Violations in Cloud Computing

The cloud service level agreement (SLA) manage the relationship between service providers and consumers in cloud computing. SLA is an integral and critical part of modern era IT vendors and communication contracts. Due to low cost and flexibility more and more consumers delegate their tasks to cloud providers, the SLA emerges as a key aspect between the consumers and providers. Continuous monitoring of Quality of Service (QoS) attributes is required to implement SLAs because of the complex nature of cloud communication. Many other factors, such as user reliability, satisfaction, and penalty on violations are also taken into account. Currently, there is no such policy of cloud SLA monitoring to minimize SLA violations. In this work, we have proposed a cloud SLA monitoring policy by dividing a monitoring session into two parts, for critical and non-critical parameters. The critical and non-critical parameters will be decided on the interest of the consumer during SLA negotiation. This will help to shape a new comprehensive SLA based Proactive Resource Allocation Approach (RPAA) which will monitor SLA at runtime, analyze the SLA parameters and try to find the possibility of SLA violations. We also have implemented an adaptive system for allocating cloud IT resources based on SLA violations and detection. We have defined two main components of SLA-PRAA i.e., (a) Handler and (b) Accounting and Billing Manager. We have also described the function of both components through algorithms. The experimental results validate the performance of our proposed method in comparison with state-of-the-art cloud SLA policies.

Incremental execution of temporal graph queries over runtime models with history and its applications

Modern software systems are intricate and operate in highly dynamic environments for which few assumptions can be made at design-time. This setting has sparked an interest in solutions that use a runtime model which reflects the system state and operational context to monitor and adapt the system in reaction to changes during its runtime. Few solutions focus on the evolution of the model over time, i.e., its history, although history is required for monitoring temporal behaviors and may enable more informed decision-making. One reason is that handling the history of a runtime model poses an important technical challenge, as it requires tracing a part of the model over multiple model snapshots in a timely manner. Additionally, the runtime setting calls for memory-efficient measures to store and check these snapshots. Following the common practice of representing a runtime model as a typed attributed graph, we introduce a language which supports the formulation of temporal graph queries, i.e., queries on the ordering and timing in which structural changes in the history of a runtime model occurred. We present a querying scheme for the execution of temporal graph queries over history-aware runtime models. Features such as temporal logic operators in queries, the incremental execution, the option to discard history that is no longer relevant to queries, and the in-memory storage of the model, distinguish our scheme from relevant solutions. By incorporating temporal operators, temporal graph queries can be used for runtime monitoring of temporal logic formulas. Building on this capability, we present an implementation of the scheme that is evaluated for runtime querying, monitoring, and adaptation scenarios from two application domains.

From LTL to rLTL monitoring: improved monitorability through robust semantics

Runtime monitoring is commonly used to detect the violation of desired properties in safety critical cyber-physical systems by observing its executions. Bauer et al. introduced an influential framework for monitoring Linear Temporal Logic (LTL) properties based on a three-valued semantics for a finite execution: the formula is already satisfied by the given execution, it is already violated, or it is still undetermined, i.e., it can still be satisfied and violated by appropriate extensions of the given execution. However, a wide range of formulas are not monitorable under this approach, meaning that there are executions for which satisfaction and violation will always remain undetermined no matter how it is extended. In particular, Bauer et al. report that 44% of the formulas they consider in their experiments fall into this category. Recently, a robust semantics for LTL was introduced to capture different degrees by which a property can be violated. In this paper we introduce a robust semantics for finite strings and show its potential in monitoring: every formula considered by Bauer et al. is monitorable under our approach. Furthermore, we discuss which properties that come naturally in LTL monitoring—such as the realizability of all truth values—can be transferred to the robust setting. We show that LTL formulas with robust semantics can be monitored by deterministic automata, and provide tight bounds on the size of the constructed automaton. Lastly, we report on a prototype implementation and compare it to the LTL monitor of Bauer et al. on a sample of examples.

VerificationTalk: A Verification and Security Mechanism for IoT Applications.

An Internet of Things (IoT) application typically involves implementations in both the device domain and the network domain. In this two-domain environment, it is possible that application developers implement the wrong network functions and/or connect some IoT devices that should never be linked, which result in the execution of wrong operations on network functions. To resolve these issues, we propose the VerificationTalk mechanism to prevent inappropriate IoT application deployment. VerificationTalk consists of two subsystems: the BigraphTalk subsystem which verifies IoT device configuration; and AFLtalk which validates the network functions. VerificationTalk provides mechanisms to conduct online anomaly detection by using a runtime monitor and offline by using American Fuzzy Lop (AFL). The runtime monitor is capable of intercepting potentially harmful data targeting IoT devices. When VerificationTalk detects errors, it provides feedback for debugging. VerificationTalk also assists in building secure IoT applications by identifying security loopholes in network applications. By the appropriate design of the IoTtalk execution engine, the testing capacity of AFLtalk is three times that of traditional AFL approaches.

Worst-case Execution Time Calculation for Query-based Monitors by Witness Generation

Runtime monitoring plays a key role in the assurance of modern intelligent cyber-physical systems, which are frequently data-intensive and safety-critical. While graph queries can serve as an expressive yet formally precise specification language to capture the safety properties of interest, there are no timeliness guarantees for such auto-generated runtime monitoring programs, which prevents their use in a real-time setting. While worst-case execution time (WCET) bounds derived by existing static WCET estimation techniques are safe, they may not be tight as they are unable to exploit domain-specific (semantic) information about the input models. This article presents a semantic-aware WCET analysis method for data-driven monitoring programs derived from graph queries. The method incorporates results obtained from low-level timing analysis into the objective function of a modern graph solver. This allows the systematic generation of input graph models up to a specified size (referred to as witness models ) for which the monitor is expected to take the most time to complete. Hence, the estimated execution time of the monitors on these graphs can be considered as safe and tight WCET. Additionally, we perform a set of experiments with query-based programs running on a real-time platform over a set of generated models to investigate the relationship between execution times and their estimates, and we compare WCET estimates produced by our approach with results from two well-known timing analyzers, aiT and OTAWA.

Hardware Trojan detection for lightweight ciphers implemented on field‐programmable gate arrays using the replay algorithm

SummaryHardware Trojans (HTs) are malicious and intentional modification of the hardware design, embedded by an adversary to leak sensitive data, modify functionality, or cause malfunction. Lightweight ciphers are designed for resource‐constrained devices (RCDs) to balance resources and security, but often targeted by HTs. This work aims to create a trusted lightweight cipher design and detect complex sequential HTs using a runtime monitoring method with minimal resource overhead. The novelty of the approach is that it combines full or partial temporal redundancy, resource minimization, and runtime monitoring. The algorithm employs temporal redundancy, by executing a first run of the cipher, and then replays the cipher fully (ReplayN method) or partially (ReplayN/2 method). A mismatch indicates HT activity, and a third replay is executed to determine the correct results. The ReplayN/2 algorithm further optimizes time and energy by limiting replay to the second half of the cipher rounds. The proposed method is compared with other existing methods including triple modular redundancy (TMR), adapted triple modular redundancy (ATMR), and reverse‐function redundancy (RFR). All methods were implemented in field‐programmable gate array (FPGA) technology. Implementation performance metrics were measured and compared, including resource utilization (i.e., logical elements [LEs]), power, energy (E), and speed (i.e., Fmax). The results show that the proposed algorithm outperforms existing methods in most of the metrics. In terms of speed and area, the replay algorithm is faster by 24% and reduces total LEs by 42% when compared with existing methods. Regarding power, replay algorithm reduces power by 45% when compared with TMR and ATMR. With respect to energy, replay methods reduce energy by an average of 6% when compared with TMR and ATMR. The ReplayN/2 reduces energy by an average 15% when compared with TMR and ATMR. Based on the (i.e., energy × area) metric, ReplayN/2 is the top performing method as it balances area and energy metrics. Using metric, the replay methods are the best performing methods as they best balance area, energy, and speed. Also, based on metric, the replay methods are the top performing methods.

Dynamic Evaluation of Microservice Granularity Adaptation

Microservices have gained acceptance in software industries as an emerging architectural style for autonomic, scalable, and more reliable computing. Among the critical microservice architecture design decisions is when to adapt the granularity of a microservice architecture by merging/decomposing microservices. No existing work investigates the following question: How can we reason about the trade-off between predicted benefits and cost of pursuing microservice granularity adaptation under uncertainty? To address this question, we provide a novel formulation of the decision problem to pursue granularity adaptation as a real options problem. We propose a novel evaluation process for dynamically evaluating granularity adaptation design decisions under uncertainty. Our process is based on a novel combination of real options and the concept of Bayesian surprises. We show the benefits of our evaluation process by comparing it to four representative industrial microservice runtime monitoring tools, which can be used for retrospective evaluation for granularity adaptation decisions. Our comparison shows that our process can supersede and/or complement these tools. We implement a microservice application—Filmflix—using Amazon Web Service Lambda and use this implementation as a case study to show the unique benefit of our process compared to traditional application of real options analysis.

Interlocking Safety Cases for Unmanned Autonomous Systems in Shared Airspaces

The growing adoption of unmanned aerial vehicles (UAVs) for tasks such as eCommerce, aerial surveillance, and environmental monitoring introduces the need for new safety mechanisms in an increasingly cluttered airspace. In our work we thus emphasize safety issues that emerge at the intersection of infrastructures responsible for controlling the airspace, and the diverse UAVs operating in their space. We build on safety assurance cases (SAC) - a state-of-the-art solution for reasoning about safety - and propose a novel approach based on interlocking SACs. The infrastructure safety case (ISAC) specifies assumptions upon UAV behavior, while each UAV demonstrates compliance to the ISAC by presenting its own (pluggable) safety case (pSAC) which connects to the ISAC through a set of interlock points. To collect information on each UAV we enforce a “trust but monitor” policy, supported by runtime monitoring and an underlying reputation model. We evaluate our approach in three ways: first by developing ISACs for two UAV infrastructures, second by running simulations to evaluate end-to-end effectiveness, and finally via an outdoor field-study with physical UAVs. The results show that interlocking SACs can be effective for identifying, specifying, and monitoring safety-related constraints upon UAVs flying in a controlled airspace.

Trace-Length Independent Runtime Monitoring of Quantitative Policies

Metric linear-time logic (MTL) has been widely used to specify runtime policies. Traditionally this use of MTL is to capture the qualitative aspects of the monitored systems, but recent developments in its extensions with aggregate operators allow some quantitative policies to be specified. Our interest in MTL-based policy languages is driven by applications in runtime malware or intrusion detection in platforms like Android and autonomous vehicles, which requires the monitoring algorithm to be independent of the length of the system event traces so that its performance does not degrade as the traces grow. We propose a policy language based on a past-time variant of MTL, extended with an aggregate operator called the metric temporal counting quantifier to specify a policy based on the number of times some sub-policies are satisfied in the specified past time interval. We show that a broad class of policies, but not all policies, specified with our language can be monitored in a trace-length independent way, and provide a concrete algorithm to do so. We implement and test our algorithm in both an existing Android monitoring framework and an autonomous vehicle simulation platform, and show that our approach can effectively specify and monitor quantitative policies drawn from real-world studies.

Towards a framework of enforcing resilient operation of cyber‐physical systems with unknown dynamics

Ensuring that safety-critical cyber-physical systems (CPSs) continue to satisfy correctness and safety specifications even under faults or adversarial attacks is very challenging, especially in the presence of legacy components for which accurate models are unknown to the designer. Current techniques for secure-by-design systems engineering do not provide an end-to-end methodology for a designer to provide real-time assurance for safety-critical CPSs by identifying system dynamics and updating control strategies in response to newly discovered faults, attacks or other changes such as system upgrades. We propose a new methodology, along with an integrated framework implemented in MATLAB to guarantee the resilient operation of safety-critical CPSs with unknown dynamics. The proposed framework consists of three main components. The runtime monitor evaluates the system behaviour on-the-fly against its correctness specifications expressed as signal temporal logic formulas. The model synthesiser incorporates a sparse identification approach that is used to continually update the plant model and control policies to adapt to any changes in the system or the environment. The decision and control module designs a controller to ensure that the correctness specifications are satisfied at runtime. For evaluation, we apply our proposed framework to ensure the resilient operations of two CPS case studies.

A Novel Spatial–Temporal Specification-Based Monitoring System for Smart Cities

With the development of the Internet of Things, millions of sensors are being deployed in cities to collect real-time data. This leads to a need for checking city states against city requirements at runtime. In this article, we develop a novel spatial-temporal specification-based monitoring system for smart cities. We first describe a study of over 1000 smart city requirements, some of which cannot be specified using the existing logic, such as the signal temporal logic (STL) and its variants. To tackle this limitation, we develop spatial aggregation STL (SaSTL)-a novel spatial aggregation STL-for the efficient runtime monitoring of safety and performance requirements in smart cities. We develop two new logical operators in SaSTL to augment STL for expressing spatial aggregation and spatial counting characteristics that are commonly found in real city requirements. We define the Boolean and quantitative semantics for SaSTL in support of the analysis of city performance across different periods and locations. We also develop efficient monitoring algorithms that can check the SaSTL requirement in parallel over multiple data streams (e.g., generated by multiple sensors distributed spatially in a city). Additionally, we build an SaSTL-based monitoring tool to support decision making of different stakeholders to specify and runtime monitor their requirements in smart cities. We evaluate our SaSTL monitor by applying it to three case studies with large-scale real city sensing data (e.g., up to 10 000 sensors in one study). The results show that SaSTL has a much higher coverage expressiveness than other spatial-temporal logics, and with a significant reduction of computation time for monitoring requirements. We also demonstrate that the SaSTL monitor improves the safety and performance of smart cities via simulated experiments.

Runtime verification of train control systems with parameterized modal live sequence charts

With the growing complexity of railway control systems, it is required to preform runtime safety checks of system executions that go beyond conventional runtime monitoring of pre-programmed safety conditions. Runtime verification is a lightweight and rigorous formal method that dynamically analyses execution traces against some formal specifications. A challenge in applying this method in railway systems is defining a suitable monitoring specification language, i.e., a language that is expressive, of reasonable complexity, and easy to understand. In this paper, we propose parameterized modal live sequence charts (PMLSCs) by introducing the alphabet of the specification into charts to distinguish between silent events and unexpected events. We further investigate the expressiveness and complexity theories of the language. In particular, we prove that PMLSCs are closed under negation and the complexity of a subclass of PMLSCs is linear, which allows the language to be used to monitor a system online. Finally, we use PMLSCs to monitor an RBC system in the Chinese high-speed railway and evaluate the performance. The experimental results show that the PMLSC has high monitoring efficiency, and can reduce false alarm rate by introducing alphabets of charts.

RML: Theory and practice of a domain specific language for runtime verification

• Simple, expressive, abstract Domain Specific Language for Runtime Verification. • Strict separation between instrumentation and specification to support reusability. • Portable specifications and interoperable monitors. • Full formal semantics, equational laws for optimization of generated monitors. • Compiler available with several running examples and benchmarks. Runtime verification (RV) is an approach to verification consisting in dynamically checking that the event traces generated by single runs of a system under scrutiny (SUS) are compliant with the formal specification of its expected correct behavior. RML (Runtime Monitoring Language) is a simple but powerful Domain Specific Language (DSL) for RV which is able to express non context-free properties. When designing RML, particular care has been taken to favor abstraction and simplicity, to better support reusability and portability of specifications and interoperability of the monitors generated from them; this is mainly achieved by decoupling the two problems of property specification and event generation, and by minimizing the available primitive constructs. The formalization and implementation of RML is based on a trace calculus with a fully deterministic rewriting semantics. The semantics of RML is defined by translation into such a calculus, which, in fact, is used as intermediate representation (IR) by the RML compiler. The effectiveness of RML and its methodological impact on RV are presented through interesting patterns that can be adapted to different contexts requiring verification of standard properties. A collection of tested examples is provided, together with benchmarks showing that the deterministic semantics and the performed dynamic optimizations based on the laws of the trace calculus significantly improve the performances of the generated monitors.