Bitmap Compression Research Articles

Oasis: An Optimal Disjoint Segmented Learned Range Filter

The learning-enhanced data structure has inspired the development of the range filter, bringing significantly better false positive rate (FPR) than traditional non-learned range filters. Its core idea is to employ piece-wise linear functions that uniformly map the entire key space into a bitmap sequentially. Nonetheless, such uniform mapping can be space-ineffective, impacting FPRs. This paper introduces Oasis, a novel learned range filter that divides the key space into disjointed intervals by excluding large empty ranges explicitly and optimally maps those unpruned intervals into a compressed bitmap. The configuration optimality in Oasis is guaranteed by a careful theoretical analysis. To enhance the versatility of Oasis, we further propose Oasis+, which integrates the design space of both learned and non-learned filters, delivering robust performance across a wide range of workloads. We evaluate the performance of both Oasis and Oasis+ when integrated into the key-value system RocksDB, using a diverse set of real-world and synthetic datasets and workloads. In RocksDB, Oasis and Oasis+ improve the performance by up to 1.4× and 6.2× when compared to state-of-the-art learned and non-learned range filters.

On the bitmap compression for joint coding and data hiding of AMBTC compressed images

On the bitmap compression for joint coding and data hiding of AMBTC compressed images

Evaluation of Receiver-Feedback Techniques for Fragmentation Over LPWANs

The Internet engineering task force (IETF) has standardized a new framework for IPv6 support over low-power wide-area networks (LPWANs), called static context header compression and fragmentation (SCHC). SCHC includes acknowledgment (ACK)-based mechanisms for reliable fragmented packet transmission. For the latter, SCHC defines a receiver-feedback technique (RFT), called compressed bitmap (CB), by which a receiver reports to the sender whether the fragments carrying a packet have been received or not. Such information is carried as ACK payload. Considering the extraordinary frame size and message rate constraints of LPWANs, ACK payload size becomes crucial. In this article, we compare the performance of CB with that of several alternative RFTs, namely, list of lost fragments (LLF), List of Deltas (LoD), and uncompressed bitmap (UB), where the latter is used as a benchmark. We evaluate the considered RFTs in terms of ACK size, number of Layer 2 (L2) frames needed to carry an ACK, and ACK Time on Air. Our analysis shows that the use of RFTs different from CB offers significant performance improvement in many scenarios. Furthermore, we provide guidance on which RFT should be used for different packet sizes, error rates, and error patterns.

An Efficient Indexing Scheme for Network Traffic Collection and Retrieval System

Historical network traffic retrieval, both at the packet and flow level, has been applied in many fields of network security, such as network traffic analysis and network forensics. To retrieve specific packets from a vast number of packet traces, it is an effective solution to build indexes for the query attributes. However, it brings challenges of storage consumption and construction time overhead for packet indexing. To address these challenges, we propose an efficient indexing scheme called IndexWM based on the wavelet matrix data structure for packet indexing. Moreover, we design a packet storage format based on the PcapNG format for our network traffic collection and retrieval system, which can speed up the extraction of index data from packet traces. Offline experiments on randomly generated network traffic and actual network traffic are performed to evaluate the performance of the proposed indexing scheme. We choose an open-source and widely used bitmap indexing scheme, FastBit, for comparison. Apart from the native bitmap compression method Word-Aligned Hybrid (WAH), we implement an efficient bitmap compression method Scope-Extended COMPAX (SECOMPAX) in FastBit for performance evaluation. The comparison results show that our scheme outperforms the selected bitmap indexing schemes in terms of time consumption, storage consumption and retrieval efficiency.

CBA-TCP: A cross-layer TCP transmission rate adaptation scheme based on the compressed block ACK bitmap of IEEE 802.11n networks

Optimizing the performance of TCP in wireless networks in a changing network condition is a highly challenging issue. The lack of information at the TCP layer about the link quality will result in inaccurate transmission rate. Therefore, TCP needs to have sufficient information about the environment and the different characteristics of the lower layers to improve the performance. To achieve this goal we proposed a cross-layer scheme which makes use of the information stored in the 802.11n compressed block acknowledged bitmap about the status of the transmitted subframes of the aggregated frame. This information is fed to the TCP transmission mechanism as a new criterion to adjust the TCP transmission rate with accordance to the aggregation level and channel quality. The simulation results show that the proposed scheme improves the TCP performance in term of throughput and packet loss.

An Effective Payload Attribution Scheme for Cybercriminal Detection Using Compressed Bitmap Index Tables and Traffic Downsampling

Payload attribution systems (PAS) are one of the most important tools of network forensics for detecting an offender after the occurrence of a cybercrime. A PAS stores the network traffic history in order to detect the source and destination pair of a certain data stream in case a malicious activity occurs on the network. The huge volume of information that is daily transferred in the network means that the data stored by a PAS must be as compact and concise as possible. Moreover, the investigation of this large volume of data for a malicious data stream must be handled within a reasonable time. For this purpose, several techniques based on storing a digest of traffic using Bloom filters have been proposed in the literature. The false positive rate of existing techniques for detecting cybercriminals is unacceptably high, i.e., many source and destination pairs are falsely determined as malicious, making it difficult to detect the true criminal. In order to ameliorate this problem, we have proposed a solution based on compressed bitmap index tables and traffic downsampling. Our analytical evaluation and experimental results show that the proposed method significantly reduces the false positive rate.

Roaring bitmaps: Implementation of an optimized software library

SummaryCompressed bitmap indexes are used in systems such as Git or Oracle to accelerate queries. They represent sets and often support operations such as unions, intersections, differences, and symmetric differences. Several important systems such as Elasticsearch, Apache Spark, Netflix's Atlas, LinkedIn's Pivot, Metamarkets' Druid, Pilosa, Apache Hive, Apache Tez, Microsoft Visual Studio Team Services, and Apache Kylin rely on a specific type of compressed bitmap index called Roaring. We present an optimized software library written in C implementing Roaring bitmaps: CRoaring. It benefits from several algorithms designed for the single‐instruction–multiple‐data instructions available on commodity processors. In particular, we present vectorized algorithms to compute the intersection, union, difference, and symmetric difference between arrays. We benchmark the library against a wide range of competitive alternatives, identifying weaknesses and strengths in our software. Our work is available under a liberal open‐source license.

SBH: Super byte-aligned hybrid bitmap compression

Bitmap indexes are commonly used in data warehousing applications such as on-line analytic processing (OLAP). Storing the bitmaps in compressed form has been shown to be effective not only for low cardinality attributes, as conventional wisdom would suggest, but also for high cardinality attributes. Compressed bitmap indexes, such as Byte-aligned Bitmap Compression (BBC), Word-Aligned Hybrid (WAH) and several of their variants have been shown to be efficient in terms of both time and space, compared to traditional database indexes. In this paper, we propose a new technique for compressed bitmap indexing, called Super Byte-aligned Hybrid (SBH) bitmap compression, which improves upon the current state-of-the-art compression schemes. In our empirical evaluation, the query processing time of SBH was about five times faster than that of WAH, while the size of its compressed bitmap indexes was retained nearly close to that of BBC.

Consistently faster and smaller compressed bitmaps with Roaring

SummaryCompressed bitmap indexes are used in databases and search engines. Many bitmap compression techniques have been proposed, almost all relying primarily on run‐length encoding (RLE). However, on unsorted data, we can get superior performance with a hybrid compression technique that uses both uncompressed bitmaps and packed arrays inside a two‐level tree. An instance of this technique, Roaring, has recently been proposed. Due to its good performance, it has been adopted by several production platforms (e.g., Apache Lucene, Apache Spark, Apache Kylin, and Druid). Yet there are cases where run‐length‐encoded bitmaps are smaller than the original Roaring bitmaps—typically when the data are sorted so that the bitmaps contain long compressible runs. To better handle these cases, we build a new Roaring hybrid that combines uncompressed bitmaps, packed arrays, and RLE‐compressed segments. The result is a new Roaring format that compresses better. Overall, our new implementation of Roaring can be several times faster (up to two orders of magnitude) than the implementations of traditional RLE‐based alternatives (WAH, Concise, and EWAH) while compressing better. We review the design choices and optimizations that make these good results possible. Copyright © 2016 John Wiley & Sons, Ltd.

Compression Research of Bitmap Data in Three-Dimensional Printing

In the paper, it makes 3-D (three dimensional) printing rapid prototyping bitmap data compression research based on bytes has been conducted, in the 3-D printing rapid prototyping system, where the bitmap data format is adopted, and related data storage space is very large. It firstly analyzes the bitmap data characteristics in 3-D printing rapid prototyping system and the commonly methods used in nondestructive bitmap compression, and it designs the bitmap data compression coding method based on bytes. The compression method has the characteristics such as high compression ratio, fast speed of compression and decompression, less time consuming, etc. Therefore, it can meet the increasing demand of data compression in the 3-D printing rapid prototyping system.

Performance evaluation of word-aligned compression methods for bitmap indices

Bitmap indices are a widely used scheme for large read-only repositories in data warehouses and scientific databases. This binary representation allows the use of bit-wise operations for fast query processing and is typically compressed using run-length encoding techniques. Most bitmap compression techniques are aligned using a fixed encoding length (32 or 64 bits) to avoid explicit decompression during query time. They have been proposed to extend or enhance word-aligned hybrid (WAH) compression. This paper presents a comparative study of four bitmap compression techniques: WAH, PLWAH, CONCISE, and EWAH. Experiments are targeted to identify the conditions under which each method should be applied and quantify the overhead incurred during query processing. Performance in terms of compression ratio and query time is evaluated over synthetic-generated bitmap indices, and results are validated over bitmap indices generated from real data sets. Different query optimizations are explored, query time estimation formulas are defined, and the conditions under which one method should be preferred over another are formalized.

Better bitmap performance with Roaring bitmaps

SummaryBitmap indexes are commonly used in databases and search engines. By exploiting bit‐level parallelism, they can significantly accelerate queries. However, they can use much memory, and thus, we might prefer compressed bitmap indexes. Following Oracle's lead, bitmaps are often compressed using run‐length encoding (RLE). Building on prior work, we introduce the Roaring compressed bitmap format: it uses packed arrays for compression instead of RLE. We compare it to two high‐performance RLE‐based bitmap encoding techniques: Word Aligned Hybrid compression scheme and Compressed ‘n’ Composable Integer Set. On synthetic and real data, we find that Roaring bitmaps (1) often compress significantly better (e.g., 2×) and (2) are faster than the compressed alternatives (up to 900× faster for intersections). Our results challenge the view that RLE‐based bitmap compression is best. Copyright © 2015 John Wiley & Sons, Ltd.

Fast Retrieval with Column Store using RLE Compression Algorithm

Column oriented database have continued to grow over the past few decades. C-Store, Vertica Monet DB and Lucid DB are popular open source column oriented database. Columnstore in a nutshell, store each attribute values belonging to same column contiguously. Since column data is uniform type therefore, there are some opportunities for storage size optimization in Column-store, many renowned compression schemes such as RLE & LZW that make use of similarity of adjacent data to compress. Good Compression can also be achieved using bitmap index by order of magnitude through the sorting. The Run Length Encoding works best for the columns of ordered data, or data with few distinct values. This ensures long runs of identical values which RLE compresses quite well. In this paper we have put an effort to build a simulation of Column-Store and applied the best bitmap compression technique RLE which further improves the retrieval time. General Terms Your general terms must be any term which can be used for general classification of the submitted material such as Pattern Recognition, Security, Algorithms et. al.

General Document Retrieval in Compact Space

Given a collection of documents and a query pattern, document retrieval is the problem of obtaining documents that are relevant to the query. The collection is available beforehand so that a data structure, called an index, can be built on it to speed up queries. While initially restricted to natural language text collections, document retrieval problems arise nowadays in applications like bioinformatics, multimedia databases, and web mining. This requires a more general setup where text and pattern can be general sequences of symbols, and the classical inverted indexes developed for words cannot be applied. While linear-space time-optimal solutions have been developed for most interesting queries in this general case, space usage is a serious problem in practice. In this article, we develop compact data structures that solve various important document retrieval problems on general text collections. More specifically, we provide practical solutions for listing the documents where a query pattern appears, together with its frequency in each document, and for listing k documents where a query pattern appears most frequently. Some of our techniques build on existing theoretical proposals, while others are new. In particular, we introduce a novel grammar-based compressed bitmap representation that may be of independent interest when dealing with repetitive sequences. Ours are the first practical indexes that use less space when the text collection is compressible. Our experimental results show that, in various real-life text collections, our data structures are significantly smaller than the most space-efficient previous solutions, using up to half the space without noticeably increasing the query time. Overall, document listing can be carried out in 10 to 40 milliseconds for patterns that appear 100 to 10,000 times in the collection, whereas top- k retrieval is carried out in k to 10 k milliseconds.

Study on a Hybrid Algorithm of Navigation Radar Echo Data Compression

With the conflict between storage space of the ship navigation radar system and large raw echoes data and files, a mixed compression algorithm of the bitmap compression + half-byte compression + Double Huffman compression is presented which is based on the characteristics of the text type data. On the basis of the original half-byte compression algorithm, the method combined with the bitmap compression and Huffman improved compression algorithm. If just using the half a byte compression algorithm, the compression ratio can only be 50%; while using the hybrid compression, the total compression ratio can be up to 75%, which save a large amount of storage resources for ship navigation system, and improve the rate of data transmission at the same time.

Relevant Window based Bitmap Compression in P2P Systems: Framework and Solution

P2P systems require neighbor peers to frequently exchange buffer-map (BM) messages for efficient content sharing and distribution, which, however, can result in considerable communication overhead. A big problem in the BMs exchanged between neighbor peers is that a lot of information in them is redundant. To reduce the redundancy, some P2P systems have adopted certain block-level compression schemes (e.g., Huffman encoding) to compress each BM in isolation. However, these schemes simply treat each BM separately and as a single block of data, which largely affects their compression efficiency. In this paper, we propose a novel relevant-window-based (RW) compression framework, which takes advantage of the correlation between sequentially exchanged BMs between neighbor peers and thus can greatly remove the redundancy in them. We accordingly design a RW-based distributed compression scheme, which can work alone or co-work well with an existing block-level compression scheme for higher compression efficiency. We prove the correctness of our scheme and derive tight upper bound on average length of compressed bitmaps by our scheme via mathematical modeling. Numerical results demonstrate that our scheme alone can achieve compression efficiency of 96.6%, which can be further increased to up to 97.1% when jointly working with a block-level compression scheme.

BAHUI

Mining high utility itemsets is one of the most important research issues in data mining owing to its ability to consider nonbinary frequency values of items in transactions and different profit values for each item. Although a number of relevant approaches have been proposed in recent years, they incur the problem of producing a large number of candidate itemsets for high utility itemsets. In this paper, the authors propose an efficient algorithm, namely BAHUI (Bitmap-based Algorithm for High Utility Itemsets), for mining high utility itemsets with bitmap database representation. In BAHUI, bitmap is used vertically and horizontally. On the one hand, BAHUI exploits a divide-and-conquer approach to visit itemset lattice by using bitmap vertically. On the other hand, BAHUI horizontally uses bitmap to calculate the real utilities of candidates. Using bitmap compression scheme, BAHUI reduces the memory usage and makes use of the efficient bitwise operation. Furthermore, BAHUI only records candidate high utility itemsets with maximal length, and inherits the pruning and searching strategies from maximal itemset mining problem. Extensive experimental results show that the BAHUI algorithm is both efficient and scalable.

A Density based Priority Queue Strategy to Evaluate Iceberg Queries Efficiently using Compressed Bitmap Indices

iceberg query is a special class of aggregation query that computes aggregated values upon user interested threshold (T).The bitmap index is a common data structure for fast retrieval of matching tuples from data base table. These resultant tuples are useful to compute aggregations such as SUM, COUNT, AVG, MIN, MAX, and RANK. In this paper, we propose a density based bitmap pruning strategy to evaluate iceberg queries efficiently using compressed bitmap indices. The strategy prioritizes the vectors to be enter in to priority queue by allowing high density of 1's count that achieve optimal pruning effect. Extensive experimentation demonstrates our proposed approach is much more efficient than existing strategy.

Enhancing the Search in MOLAP Sparse Data

Multidimensional on-line analytical processing (MOLAP) systems deal well with dense data than relational ones (ROLAP). In the existence of sparse data, MOLAP systems become memory consuming, which may limit and slow down data processing tasks. Many compression techniques have been proposed to deal with the sparsity of data in MOLAP systems. One of these techniques is the bitmap compression, which allows a significant reduction of the memory space used for data processing. In this article, we propose an extension to the bitmap compression technique by storing the compressed data as bits into multiple efficient data structures based on a new indexing strategy instead of the linear structure. Compared with the classical bitmap, the proposed enhancement not only allows space reduction but also reduces the search time through the compressed data. We present some algorithms that allow maintaining and searching within the compressed structure without the need for decompression. We demonstrate that the complexity of the proposed algorithms varies from logarithmic to constant, compared with the linear complexity of the classical bitmap technique.

A Tailored Anti-Forensic Approach for Bitmap Compression in Medical Images

A Tailored Anti-Forensic Approach for Bitmap Compression in Medical Images