Abstract
Online monitoring is the task of identifying complex temporal patterns while incrementally processing streams of events. Existing state-of-the-art monitors can process streams of modest velocity in real-time: a few thousands events per second. We scale up monitoring to higher velocities by slicing the stream, based on the events’ data values, into substreams that can be independently monitored. Because monitoring is not data parallel in general, slicing can lead to data duplication. To reduce this overhead, we adapt hash-based partitioning techniques from databases to the monitoring setting. We implement the resulting automatic data slicer in Apache Flink and use the MonPoly tool to monitor the substreams. We empirically evaluate this setup, demonstrating a substantial scalability improvement.
Highlights
In large-scale software systems, millions of events occur each second [25,41]
The optimality of the hypercube approach in terms of a balanced data distribution is out of reach for general metric first-order temporal logic (MFOTL) formulas, we demonstrate that our automatic splitting results in balanced slices and improved monitoring performance
Algorithm 1’s complexity is bounded by O(|φ| · 2n · n · p), where |φ| is the size of the formula φ and n is the number of free variables in φ
Summary
In large-scale software systems, millions of events occur each second [25,41]. Identifying instances of interesting patterns in these high-velocity data streams is a central challenge in the area of runtime verification and monitoring. Often, this search must be performed online given the systems’ continuous operation and the massive amounts of data they produce. An online monitor takes as input a pattern and a stream of data, which it consumes incrementally, and it detects and outputs matches with the pattern. Srd-an Krsticis supported by the Swiss National Science Foundation Grant “Big Data Monitoring” (167162)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
