Faster Multidimensional Data Queries on Infrastructure Monitoring Systems

Abstract

Performance monitoring systems collect and analyze data, and provide insight on availability, performance, and capacity from various monitored targets. However, the analytics in online performance monitoring systems have often been limited due to the query performance of large scale multidimensional data. The existing approaches for preference or ranking queries generally include approaches with and without pre-processing. The approaches without pre-processing work well with small data sets. For larger data sets, the preference query is usually slow or not time-feasible. In this paper, we introduce and evaluate an optimized query approach for performance monitoring systems using the bit-sliced index (BSI). The objective is to enable users to perform fast interactive queries with various querying criteria and dynamic grouping on multiple dimensions. The experiments cover multidimensional grouping and preference top-k queries on the proposed approach with BSI, an approach using bitmap indexing for top-k queries, and the sequential scan sort top-k selection algorithms. The evaluation covers the single attribute query, multiple attributes weighted sum query, and multidimensional grouping using a real-time performance monitoring system data with several hundred thousand records. Working with a single attribute top-k evaluation, the proposed bit-sliced approach outperforms the bitmap approach and the sequential scan approach by a factor of 3 and a factor of 100+ respectively. Similarly, with multiple attributes sum top-k evaluation, it is 10 times faster than the sequential scan approach as well. On the multidimensional grouping, the proposed bit-sliced approach outperforms the collection stream approach by a factor of 10. In conclusion, the bit-sliced approach has faster performance than other evaluated approaches on multidimensional grouping top-k queries.