Abstract
Online monitoring is the task of identifying complex temporal patterns while incrementally processing streams of data-carrying events. Existing state-of-the-art monitors for first-order patterns, which may refer to and quantify over data values, can process streams of modest velocity in real-time. We show how to scale up first-order monitoring to substantially higher velocities by slicing the stream, based on the events’ data values, into substreams that can be monitored independently. Because monitoring is not embarrassingly parallel in general, slicing can lead to data duplication. To reduce this overhead, we adapt hash-based partitioning techniques from databases to the monitoring setting. We implement these techniques in an automatic data slicer based on Apache Flink and empirically evaluate its performance using two tools—MonPoly and DejaVu—to monitor the substreams. Our evaluation attests to substantial scalability improvements for both tools.
Highlights
In large-scale software systems, millions of events occur each second [25,41]
The optimality of the hypercube approach in terms of a balanced data distribution is out of reach for general metric first-order temporal logic (MFOTL) formulas, we demonstrate that our automatic splitting results in balanced slices and improved monitoring performance
Algorithm 1’s complexity is bounded by O(|φ| · 2n · n · p), where |φ| is the size of the formula φ and n is the number of free variables in φ
Summary
In large-scale software systems, millions of events occur each second [25,41]. Identifying instances of interesting patterns in these high-velocity data streams is a central challenge in the area of runtime verification and monitoring. Often, this search must be performed online given the systems’ continuous operation and the massive amounts of data they produce. An online monitor takes as input a pattern and a stream of data, which it consumes incrementally, and it detects and outputs matches with the pattern. Srd-an Krsticis supported by the Swiss National Science Foundation Grant “Big Data Monitoring” (167162)
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: International Journal on Software Tools for Technology Transfer
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
