Bitmap Index Research Articles

Efficient Algorithm Based on Itemset-lattice and Bitmap Index for Finding Frequent Itemsets

Abstract Finding frequent itemsets is the key problem in association rules mining. A new algorithm based on the lattice theory and bitmap index for mining frequent itemsets is proposed. The algorithm converts the original transaction database to an itemset-lattice in the pre-processing, where each itemset vertex has a label to save its support, and the complicated task of mining frequent itemsets in the database is thus changed to the simpler one that searches vertexes in the lattice. The efficiency of the mining process is enhanced greatly. Since the support counting in the association rules mining incurs a high cost regarding the I/O, a bitmap index technique is used to speed up the counting process. To address the issue that the intact bitmap usually needs a big memory space for storage, the bit vector is partitioned into blocks, which can be encoded as a symbol. This makes the original bitmap more compact in storage and improve the support counting efficiency as well. Finally, experimental and analytical results are presented.

Spatial Data Management and Analysis System for Flood Hazard Mitigation of Poyang Lake Watershed, China

Flood hazard Prevention and mitigation of Poyang Lake watershed is an emergent environmental problem because Poyang Lake watershed is one of the regions exposed most frequently to flood hazard in China. In order to meet the need of disaster reduction decision making of Poyang Lake watershed, Spatial Data Management and Analysis System of China Land Territory (not include islands in South Sea of China)for Flood Hazard Mitigation(short for SMAS-FHM) has been established making use of the methodology of spatial database technology based on Oracle9i&ArcSDE9. This paper, probes into designing and implementation of multi-resources and scales massive spatial database management system about flood hazard, and integral management between hazard thematic data and spatial data. Especially, a bitmap index adapted to primal TM/ETM+ image has been put forwards and realized to accelerate data search in the spatial database, which is turned out to be efficient and effective. Based on carrying out experiment, this paper has been also researched the key issues, including massive spatial data and flood hazard thematic data organization and storage, spatial index creating, spatial data searching, and flood hazard analysis models realizing and their effective organizing, such as flooding areas computing, 3-dimension flooding simulation, refuge location selection, refugee evacuation path analysis and relief material dispensing. Finally, important applications of SMAS-FHM in flood hazard mitigation have been also discussed for Poyang Lake watershed.

Multi-resolution bitmap indexes for scientific data

The unique characteristics of scientific data and queries cause traditional indexing techniques to perform poorly on scientific workloads, occupy excessive space, or both. Refinements of bitmap indexes have been proposed previously as a solution to this problem. In this article, we describe the difficulties we encountered in deploying bitmap indexes with scientific data and queries from two real-world domains. In particular, previously proposed methods of binning, encoding, and compressing bitmap vectors either were quite slow for processing the large-range query conditions our scientists used, or required excessive storage space. Nor could the indexes easily be built or used on parallel platforms. In this article, we show how to solve these problems through the use of multi-resolution, parallelizable bitmap indexes, which support a fine-grained trade-off between storage requirements and query performance. Our experiments with large data sets from two scientific domains show that multi-resolution, parallelizable bitmap indexes occupy an acceptable amount of storage while improving range query performance by roughly a factor of 10, compared to a single-resolution bitmap index of reasonable size.

An efficient bit-based feature selection method

Feature selection is about finding useful (relevant) features to describe an application domain. Selecting relevant and enough features to effectively represent and index the given dataset is an important task to solve the classification and clustering problems intelligently. This task is, however, quite difficult to carry out since it usually needs a very time-consuming search to get the features desired. This paper proposes a bit-based feature selection method to find the smallest feature set to represent the indexes of a given dataset. The proposed approach originates from the bitmap indexing and rough set techniques. It consists of two-phases. In the first phase, the given dataset is transformed into a bitmap indexing matrix with some additional data information. In the second phase, a set of relevant and enough features are selected and used to represent the classification indexes of the given dataset. After the relevant and enough features are selected, they can be judged by the domain expertise and the final feature set of the given dataset is thus proposed. Finally, the experimental results on different data sets also show the efficiency and accuracy of the proposed approach.

Efficient online mining of large databases

Great efforts have been achieved to apply data mining algorithms onto large databases. However, long processing times remain a practical issue. This paper presents a framework to offer to database users online operators for mining large databases without size limit, in acceptable processing times. First, we integrate decision tree algorithms directly into database management systems. We are thus only limited by disc capacity and not by main memory. However, disc accesses still induce long response times. Hence, we propose two optimisations in a second step: reducing the size of the learning database by building its corresponding contingency table and reducing the number of database accesses by exploiting bitmap indices. Thus, the various decision tree based methods we implemented within Oracle deal with contingency tables or bitmap indices rather than with the whole training set. Experimentations performed show the efficiency of our integrated methods.

Presto authorization: a bitmap indexing scheme for high-speed access control to XML documents

XML (extensible markup language) is becoming the de facto standard for information exchange over the Internet. As more and more sensitive information gets stored in the form of XML, proper access control to the XML documents becomes increasingly important. However, traditional access control methodologies that have been adapted for XML documents do not address the performance issue of access control. This paper proposes a bitmap-indexing scheme in which access control decisions can be sped up. Authorization policies of the form (subject, object, and action) are encoded as bitmaps in the same manner as XML document indexes are constructed. These two are then efficiently pipelined and manipulated for fast access control and secure retrieval of XML documents.

Optimizing bitmap indices with efficient compression

Bitmap indices are efficient for answering queries on low-cardinality attributes. In this article, we present a new compression scheme called Word-Aligned Hybrid (WAH) code that makes compressed bitmap indices efficient even for high-cardinality attributes. We further prove that the new compressed bitmap index, like the best variants of the B-tree index, is optimal for one-dimensional range queries. More specifically, the time required to answer a one-dimensional range query is a linear function of the number of hits. This strongly supports the well-known observation that compressed bitmap indices are efficient for multidimensional range queries because results of one-dimensional range queries computed with bitmap indices can be easily combined to answer multidimensional range queries. Our timing measurements on range queries not only confirm the linear relationship between the query response time and the number of hits, but also demonstrate that WAH compressed indices answer queries faster than the commonly used indices including projection indices, B-tree indices, and other compressed bitmap indices.

Bitmap indices for fast end-user physics analysis in ROOT

Most physics analysis jobs involve multiple selection steps on the input data. These selection steps are called cuts or queries. A common strategy to implement these queries is to read all input data from files and then process the queries in memory. In many applications the number of variables used to define these queries is a relative small portion of the overall data set therefore reading all variables into memory takes unnecessarily long time. In this paper we describe an integration effort that can significantly reduce this unnecessary reading by using an efficient compressed bitmap index technology. The primary advantage of this index is that it can process arbitrary combinations of queries very efficiently, while most other indexing technologies suffer from the “curse of dimensionality” as the number of queries increases. By integrating this index technology with the ROOT analysis framework, the end-users can benefit from the added efficiency without having to modify their analysis programs. Our performance results show that for multi-dimensional queries, bitmap indices outperform the traditional analysis method up to a factor of 10.

A novel feature selection method for large-scale data sets1

Feature selection is about finding useful (relevant) features to describe an application domain. The problem of finding the minimal subsets of features that can describe all of the concepts in the given data set is NP-hard. In the past, we had proposed a feature selection method, which originated from rough set and bitmap indexing techniques, to select the optimal (minimal) feature set for the given data set efficiently. Although our method is sufficient to guarantee a solution's optimality, the computation cost is very high when the number of features is huge. In this paper, we propose a nearly optimal feature selection method, called bitmap-based feature selection method with discernibility matrix, which employs a discernibility matrix to record the important features during the construction of the cleansing tree to reduce the processing time. And the corresponding indexing and selecting algorithms for such feature selection method are also proposed. Finally, some experiments and comparisons are given and the result shows the efficiency and accuracy of our proposed method.

FastBit: an efficient indexing technology for accelerating data-intensive science

FastBit is a software tool for searching large read-only datasets. It organizes user data in a column-oriented structure which is efficient for on-line analytical processing (OLAP), and utilizes compressed bitmap indices to further speed up query processing. Analyses have proven the compressed bitmap index used in FastBit to be theoretically optimal for onedimensional queries. Compared with other optimal indexing methods, bitmap indices are superior because they can be efficiently combined to answer multi-dimensional queries whereas other optimal methods can not. In this paper, we first describe the searching capability of FastBit, then briefly highlight two applications that make extensive use of FastBit, namely Grid Collector and DEX.

Dynamic query scheduling in parallel data warehouses

AbstractParallel processing is a key to high performance in very large data warehouse applications that execute complex analytical queries on huge amounts of data. Although parallel database systems (PDBSs) have been studied extensively in the past decades, the specifics of load balancing in parallel data warehouses have not been addressed in detail.In this study, we investigate how the load balancing potential of a Shared Disk (SD) architecture can be utilized for data warehouse applications. We propose an integrated scheduling strategy that simultaneously considers both processors and disks, regarding not only the total workload on each resource but also the distribution of load over time. We evaluate the performance of the new method in a comprehensive simulation study and compare it to several other approaches. The analysis incorporates skew aspects and considers typical data warehouse features such as star schemas with large fact tables and bitmap indices. Copyright © 2003 John Wiley & Sons, Ltd.

A mapping mechanism to support bitmap index and other auxiliary structures on tables stored as primary B + -trees

Any auxiliary structure, such as a bitmap or a B + -tree index, that refers to rows of a table stored as a primary B + -tree (e.g., tables with clustered index in Microsoft SQL Server, or index-organized tables in Oracle) by their physical addresses would require updates due to inherent volatility of those addresses. To address this problem, we propose a mapping mechanism that 1) introduces a single mapping table, with each row holding one key value from the primary B + -tree, as an intermediate structure between the primary B + -tree and the associated auxiliary structures, and 2) augments the primary B + -tree structure to include in each row the physical address of the corresponding mapping table row. The mapping table row addresses can then be used in the auxiliary structures to indirectly refer to the primary B + -tree rows. The two key benefits are: 1) the mapping table shields the auxiliary structures from the volatility of the primary B + -tree row addresses, and 2) the method allows reuse of existing conventional table mechanisms for supporting auxiliary structures on primary B + - trees.This paper presents the mapping mechanism, its possible application in supporting various auxiliary structures on primary B + -trees, and a case study describing its use for supporting bitmap indexes on index-organized tables in Oracle9 i . The case study demonstrates that the proposed mapping mechanism allows us to reuse existing bitmap index technologies with minimal changes. It also includes a comparison between bitmap and non-bitmap (B + - tree) index performance on index-organized tables for both single-table queries and star queries. The analytical and experimental studies show that the method is storage efficient, and (despite the mapping table overhead) provides performance benefits that are similar to those provided by bitmap indexes implemented on conventional tables.

TREE BASED INDEXES VERSUS BITMAP INDEXES: A PERFORMANCE STUDY

Data warehouses are used to store large amounts of data which is often used for On-Line Analytical Processing (OLAP). Short response times are essential for on-line decision support. Common approaches to reach this goal in read-mostly environments are the precomputation of materialized views and the use of index structures. This paper focuses on the use of index structures for supporting fast access to data. The performance of index structures depends on many different parameters. Here, we focus on a set of nine parameters. Two approaches are presented to support the decision making process which index structure should be applied. The first approach is based on classification trees. The second approach uses an aggregation and scatter diagram method. Both approaches are applied to four distinct index structures: a tree-based index structure without aggregated data, a tree-based index structure with aggregated data and two bitmap index structures. This paper presents results of the comparison with both approaches.

BitCube: A Three-Dimensional Bitmap Indexing for XML Documents

XML is a new standard for exchanging and representing information on the Internet. Documents can be hierarchically represented by XML-elements. In this paper, we propose that an XML document collection be represented and indexed using a bitmap indexing technique. We define the similarity and popularity operations suitable for bitmap indexes. We also define statistical measurements in the BitCube: center, and radius. Based on these measurements, we describe a new bitmap indexing based technique to cluster XML documents. The techniques for clustering are motivated by the fact that the bitmap indexes are expected to be very sparse. Furthermore, a 2-dimensional bitmap index is extended to a 3-dimensional bitmap index, called the BitCube. Sophisticated querying of XML document collections can be performed using primitive operations such as slice, project, and dice. Experiments show that the BitCube can be created efficiently and the primitive operations can be performed more efficiently with the BitCube than with other alternatives.

Indexing images in Oracle8i

Content-based retrieval of images is the ability to retrieve images that are similar to a query image. Oracle8i Visual Information Retrieval provides this facility based on technology licensed from Virage, Inc. This product is built on top of Oracle8i interMedia which enables storage, retrieval and management of images, audios and videos. Images are matched using attributes such as color, texture and structure and efficient content-based retrieval is provided using indexes of an image index type. The design of the index type is based on a multi-level filtering algorithm. The filters reduce the search space so that the expensive comparison algorithm operates on a small subset of the data. Bitmap indexes are used to evaluate the first filter resulting in a design which performs well and is scalable. The image index type is built using Oracle8i extensible indexing technology, allowing users to create, use, and drop instances of this index type as they would any other standard index. In this paper we present an overview of the product, the design of the image index type, and some performance results of our product.

Applying on-line bitmap indexing to reduce counting costs in mining association rules

Counting costs have a great impact on efficiency of mining association rules in a large database of sales transactions. In this paper, we first formally analyze the facts that determine counting costs. Secondly, we present an on-line bitmap indexing technique to speed-up the counting process. Besides theoretical analysis, our implementation reports suggest that this indexing technique may reduce counting costs up to 50%, and almost at no costs.

An efficient bitmap encoding scheme for selection queries

Bitmap indexes are useful in processing complex queries in decision support systems, and they have been implemented in several commercial database systems. A key design parameter for bitmap indexes is the encoding scheme , which determines the bits that are set to 1 in each bitmap in an index. While the relative performance of the two existing bitmap encoding schemes for simple selection queries of the form “ v 1 ≤ A ≤ v 2 ” is known (specifically, one of the encoding schemes is better for processing equality queries; i.e., v 1 = v 2 , while the other is better for processing range queries; i.e., v 1 < v 2 ), it remains an open question whether these two encoding schemes are indeed optimal for their respective query classes in the sense that there is no other encoding scheme with better space-time tradeoff. In this paper, we establish a number of optimality results for the existing encoding schemes; in particular, we prove that neither of the two known schemes is optimal for the class of two-sided range queries. We also propose a new encoding scheme and prove that it is optimal for that class. Finally, we present an experimental study comparing the performance of the new encoding scheme with that of the existing ones as well as four hybrid encoding schemes for both simple selection queries and the more general class of membership queries of the form “ A ∈ { v 1 , v 2 , .…, v k }”. These results demonstrate that the new encoding scheme has an overall better space-time performance than existing schemes.

Indexing medium-dimensionality data in Oracle

With the advent of GIS, multi-media, and warehousing technologies, database systems have started focusing on storage and access of multi-dimensional data such as spatial, OLAP, image, audio, and video attributes. As a step in this direction, Oracle% launched the interMedia product to support spatial and image data, and Materialized Views (MV) to support warehousing applications. Although 2-dimensional spatial data is efficiently indexed using OracleBi Spatial, and highdimensional image data using a combination of bitmap indexing and the Visual Information Retrieval (VIR) product, there is still a need for efficient indexing mechanisms for medium-dimensionality data such as OLAP, and CAD/CAM applications. In this paper, we describe the implementation of a new indextype, called the R-tree, to support medium-dimensionality data (i.e., data whose dimensionality is in the range of 310) using the extensible indexing framework [DDSS95] of OracleSi. This indextype combines some of the best features of existing R-tree variants [Gut84, BKSSSO, WJ96, LLE97, RKV95].

Query optimization for selections using bitmaps

Bitmaps are popular indexes for data warehouse (DW) applications and most database management systems offer them today. This paper proposes query optimization strategies for selections using bitmaps. Both continuous and discrete selection criteria are considered. Query optimization strategies are categorized into static and dynamic. Static optimization strategies discussed are the optimal design of bitmaps, and algorithms based on tree and logical reduction. The dynamic optimization discussed is the approach of inclusion and exclusion for both bit-sliced indexes and encoded bitmap indexes.

Bitmap index design and evaluation

Bitmap indexing has been touted as a promising approach for processing complex adhoc queries in read-mostly environments, like those of decision support systems. Nevertheless, only few possible bitmap schemes have been proposed in the past and very little is known about the space-time tradeoff that they offer. In this paper, we present a general framework to study the design space of bitmap indexes for selection queries and examine the disk-space and time characteristics that the various alternative index choices offer. In particular, we draw a parallel between bitmap indexing and number representation in different number systems, and define a space of two orthogonal dimensions that captures a wide array of bitmap indexes, both old and new. Within that space, we identify (analytically or experimentally) the following interesting points: (1) the time-optimal bitmap index; (2) the space-optimal bitmap index; (3) the bitmap index with the optimal space-time tradeoff (knee); and (4) the time-optimal bitmap index under a given disk-space constraint. Finally, we examine the impact of bitmap compression and bitmap buffering on the space-time tradeoffs among those indexes. As part of this work, we also describe a bitmap-index-based evaluation algorithm for selection queries that represents an improvement over earlier proposals. We believe that this study offers a useful first set of guidelines for physical database design using bitmap indexes.