Accelerate bitmap indexing construction with massive scientific data

Abstract

The volume of scientific data has been increasing dramatically over the past decades. Due to its large data size, it becomes rather hard to efficiently organize the archived scientific data so as to provide fast data access capability. The indexing techniques, especially the multidimensional indexing techniques which can handle complex data, are heavily demanded and thus have been playing a critical role in many modern scientific applications. Bitmap index is a type of technique which aims to efficiently organize multidimensional data objects for fast data access. Modern scientific data often involve massive data which are stored in a distributed cluster, while most of the existing bitmap indexing approaches are designed for single server. Therefore, there are still many technical challenges in big scientific data based on bitmap indexing. In this paper, aiming at accelerating the bitmap index construction, we explore a methodology to efficiently build the representative bitmap technique, e.g., FastBit, in a distributed environment. A series of evaluations have been conducted over large real astronomical data sets. Those evaluations verify that our approach outperformed the conventional FastBit index construction in both small and large data sizes.