DLBench+: A benchmark for quantitative and qualitative data lake assessment

Abstract

In the last few years, the concept of data lake has become trendy for data storage and analysis. Thus, several approaches have been proposed to build data lake systems. However, such proposals are difficult to evaluate as there are no commonly shared criteria for comparing data lake systems. Thus, we introduce in this paper DLBench+, a benchmark to evaluate and compare data lake implementations that support textual and/or tabular contents. More concretely, we propose a data model made of both textual and CSV documents, a workload model composed of a set of various tasks, as well as a set of performance-based metrics, all relevant to the context of data lakes. Beyond a purely quantitative assessment, we also propose a methodology to qualitatively evaluate data lake systems through the assessment of user experience. As a proof of concept, we use DLBench+ to evaluate an open source data lake system we developed.