SECOMPAX: A bitmap index compression algorithm

Abstract

Archiving of Internet traffic is essential for analyzing network events in the field of network security. Currently, bitmap indexing is used to accelerate the indexing and search queries for archival traffic data. However, the generation of bitmap index needs large storage space, which makes bitmap index compression is a must-have function. In this paper, we propose a new bitmap index encoding algorithm named SECOMPAX (Scope-Extended COMPressed Adaptive indeX), which performs better compression ratio and fast encoding speed compared with the state-of-art bitmap index compression algorithm WAH (Word-Aligned-Hybrid), PLWAH(Position list word aligned hybrid ) and COMPAX (COMPressed Adaptive indeX). The comparison among WAH, PLWAH, COMPAX and SECOMPAX shows that SECOMAX accomplishes the smallest bitmap index in size and the comparable encoding time with other three methods. We also use real Internet trace from CAIDA to prove the validity of SECOMPAX. SECOMPAX has the best compression ratio in compared with other bitmap index encoding algorithms in our experiments. The encoding time is measured, and statistics of the distribution of codeword used in SECOMPAX is also investigated in experiments. It shows that SECOMPAX‘s extra time consumption is acceptable as the new designed codebook work effectively in encoding bit sequence which cannot be compressed in other bitmap encoding schemes. Keywords—Big Data; Network Forensic; Network Security; Bitmap Index; Index Encoding; Index Compression; WAH; COMPAX; SECOMPAX