Bench-Ranking: A First Step Towards Prescriptive Performance Analyses For Big Data Frameworks

Abstract

Leveraging Big Data (BD) processing frameworks to process large-scale Resource Description Framework (RDF) datasets holds a great interest in optimizing query performance. Modern BD services are complicated data systems, where tuning the configurations notably affects the performance. Benchmarking different frameworks and configurations provides the community with best practices towards selecting the most suitable configurations. However, most of these benchmarking efforts are classified as descriptive or diagnostic analytics. Moreover, there is no standardization for comparing and contrasting these benchmarks based on quantitative ranking techniques. This paper aims to fill this timely research gap by proposing ranking criteria (called Bench-ranking) that provide prescriptive analytics via ranking functions. In particular, Bench-ranking starts by describing the current state-of-the-art single-dimensional ranking limitations. Next, we discuss the recent benchmarking requirements for sophisticated approaches over multi-dimensional ranking. Finally, we discuss the ranking criteria goodness by reviewing its conformance and coherence metrics. We validate Bench-ranking by conducting an empirical study using large RDF datasets under a relational BD engine, i.e., Apache Spark-SQL. The proposed ranking techniques provide the practitioners with clear insights to make an informed decision, especially with experimental trade-offs for such complex solution space.