A multi-start simulated annealing strategy for Data Lake Organization Problem

Abstract

The Data Lake Organization Problem consists of optimized data navigation structures generation to reduce the user’s time exploring all available data. The goal is to find a data organization that maximizes the expected probability of table discovery during user navigation. For this problem, we propose a simulated annealing metaheuristic and compare it with the Organize literature solution on benchmark instances. The instances are Socrata Open Data Lake samples with varying topics and open data from government entities worldwide. To validate our proposal, we performed a statistical analysis using a non-parametric test, which confirmed the dominance of our proposition over the state-of-the-art. Our proposal was more efficient and increased the expected probability of table discovery up to 15%. Thus, our strategy can find better solutions in the benchmarks evaluated even without exhaustively analyzing all of them and more effectively exploring the space of solutions.