Large-scale Data Warehouse Research Articles

Synchronous CR-OLAP Tool for Efficient Parallel Computing

Real time OLAP , or RTOLAP, is the capability to quickly retrieve, aggregate, analyze and present multidimensional data for cubes whenever there are changes to the data in the relational data sources, without having to run heavy processing on the cube. A big advantage of real time OLAP is that it calculates all relevant data and provides immediate output. One of the main roles of an RTOLAP system is that data is stored directly in main memory, or in an in memory database, enabling quicker access to the data. Another factor affecting the speed of calculation is compression data is compressed, in such a way that it can be accessed must faster in its compressed form. Additionally, pre-calculated values are not stored, therefore avoiding “data explosion”. In contrast to queries for online transaction processing (OLTP) system that typically access only a small portion of a database, OLAP queries may need to aggregate large portion of a database which often leads to performance issues. In this paper introduced CR-OLAP, a cloud based Real Time OLAP system based on a new distributed index structure for OLAP, the distributed PDCR tree, that utilizes a cloud infrastructure consisting of (m+1) multicore processors. With increasing database size, CROLAP dynamically increases m to maintain performance. The distributed PDCR tree data structure supports multiple dimension hierarchies and efficient query processing on the sophisticated dimension hierarchies which are so central to OLAP system. It is particularly efficient for complex OLAP queries that need to aggregate large portions of the data warehouses. The static data cube approach proposed by Gray et.al. and materialize all or a subset of the cuboids of the data cube in order to ensure adequate query performance. Practitioners have called for some time for a real-time OLAP approach where the OLAP system gets updated instantaneously as new data arrives and always provides an up-to-date data warehouse for the decision support process. However, a major problem for real-time OLAP is the significant performance issues with large scale data warehouses. The main aim of our research is to address these problems through the use of efficient parallel computing methods. In this paper proposed a distributed data structure for real time OLAP. To our knowledge, the real-time OLAP system that has been parallelized and optimized for contemporary multi-core architectures allows for multiple insert and multiple query transactions to be executed in parallel and in real-time.

Parallel Real-Time OLAP on Multi-Core Processors

One of the most powerful and prominent technologies for knowledge discovery in decision support systems is online analytical processing (OLAP). Most of the traditional OLAP research, and most of the commercial systems, follow the static data cube approach proposed by Gray et.al. and materialize all or a subset of the cuboids of the data cube in order to ensure adequate query performance. Practitioners have called for some time for a real-time OLAP approach where the OLAP system gets updated instantaneously as new data arrives and always provides an up-to-date data warehouse for the decision support process. However, a major problem for real-time OLAP is the significant performance issues with large scale data warehouses. The aim of our research is to address these problems through the use of efficient parallel computing methods. In this paper, we present a parallel real-time OLAP system for multi-core processors. To our knowledge, this is the first real-time OLAP system that has been parallelized and optimized for contemporary multi-core architectures. Our system allows for multiple insert and multiple query transactions to be executed in parallel and in real-time. We evaluated our method for a multitude of scenarios (different ratios of insert and query transactions, query transactions with different amounts of data aggregation, different database sizes, etc.), using the TPCDS “Decision Support” benchmark data set. As multi-core test platforms, we used an Intel Sandy Bridge processor with 4 cores (8 hardware supported threads) and an Intel Xeon Westmere processor with 20 cores (40 hardware supported threads). The tests demonstrate that, with increasing number of processor cores, our parallel system achieves close to linear speedup in transaction response time and transaction throughput. On the 20 core architecture we achieved, for a 100 GB database, a better than 0.25 second query response time for real-time OLAP queries that aggregate 25% of the database. Since hardware performance improvements are currently, and in the foreseeable future, achieved not by faster processors but by increasing the number of processor cores, our new parallel real-time OLAP method has the potential to enable OLAP systems that operate in real-time on large databases.

D-Apriori: An Algorithm to Incorporate Dynamism in Apriori Algorithm

Apriori algorithm mines the data from the large scale data warehouse using association rule mining. In this paper a new algorithm named as Dynamic Apriori (D-Apriori) algorithm is presented. The proposed D-Apriori algorithm incorporates the dynamism in classical Apriori for efficiently mining the frequent itemsets from a large scale database. With the help of experimental results, it is shown that the D-Apriori algorithm performs better than the existing Apriori algorithm with respect to execution time for the dynamic behavior of data itemset.

A Context-Based Performance Enhancement Algorithm for Columnar Storage in MapReduce with Hive

To achieve high reliability and scalability, most large-scale data warehouse systems have adopted the cluster-based architecture. In this context, MapReduce has emerged as a promising architecture for large scale data warehousing and data analytics on commodity clusters. The MapReduce framework offers several lucrative features such as high fault-tolerance, scalability and use of a variety of hardware from low to high range. But these benefits have resulted in substantial performance compromise. In this paper, we propose the design of a novel cluster-based data warehouse system, Daenyrys for data processing on Hadoop – an open source implementation of the MapReduce framework under the umbrella of Apache. Daenyrys is a data management system which has the capability to take decision about the optimum partitioning scheme for the Hadoop's distributed file system (DFS). The optimum partitioning scheme improves the performance of the complete framework. The choice of the optimum partitioning is query-context dependent. In Daenyrys, the columns are formed into optimized groups to provide the basis for the partitioning of tables vertically. Daenyrys has an algorithm that monitors the context of current queries and based on the observations, it re-partitions the DFS for better performance and resource utilization. In the proposed system, Hive, a MapReduce-based SQL-like query engine is supported above the DFS.

Online expansion of large-scale data warehouses

Modern data warehouses store exceedingly large amounts of data, generally considered the crown jewels of an enterprise. The amount of data maintained in such data warehouses increases significantly over time---often at a continuous pace, e.g., by gathering additional data or retaining data for longer periods to derive additional business value, but occasionally also precipitously, e.g., when consolidating disparate data warehouses and Data Marts into a single database. Having to expand a data warehouse with 100's of TB of data by a substantial portion, e.g., 100% or more is a complex and disruptive maintenance operation as it typically involves some sort of dumping and reloading of data which requires substantial downtime. In this paper we describe the methodology and mechanisms we developed in Greenplum Database to expand large-scale data warehouses in an online fashion, i.e., without noticeable downtime. At the core of our approach is a set of robust and transactionally consistent primitives that enable efficient data movement. Special emphasis was put on usability and control that lets an administrator tailor the expansion process to specific operational characteristics via priorities and schedules. We present a number of experiments to quantify the impact of an on-going expansion on query workloads.

Data Mining in the Research of Corporate Acquisition Performance

With a descriptive and predictive ability, data mining can discover the hidden patterns of data from a large-scale data warehouse by precise mathematical means, which is valuable for decision making, especially for corporate top management. With a regression analysis based on a large survey data, this paper examines an important issue concerned to corporate acquisition. It is found that acquisition performance is positively influenced by organizational adjustment, top management turnover, buyers’ experience, target firm’s performance, and business relevance, and negatively influenced by region distance, shareholding ratio, state-owned shares of buyers, and relative size. The founding can have some theoretical value for corporate decision on the process of acquisitions and how to enhance acquisition performance and also indicate the practical value of data mining technology.

Data Warehousing Requirements Collection and Definition

Defining data warehouse requirements is widely recognized as one of the most important steps in the larger data warehouse system development process. This paper examines the potential risks and pitfalls within the data warehouse requirement collection and definition process. A real scenario of a large-scale data warehouse implementation is given, and details of this project, which ultimately failed due to inadequate requirement collection and definition process, are described. The presented case underscores and illustrates the impact of the requirement collection and definition process on the data warehouse implementation, while the case is analyzed within the context of the existing approaches, methodologies, and best practices for prevention and avoidance of typical data warehouse requirement errors and oversights.

Reliability Design for Large Scale Data Warehouses

Data reliability has been drawn much concern in large-scale data warehouses with 1PB or more data. It highly depends on many inter-dependent system parameters, such as the replica placement policies, number of nodes and so on. Previous work has roughly and separately discussed the individual impacts of these parameters, and seldom provided their optimal values, nor mentioned their optimal combination. In this paper, we present a new object-based-repairing Markov model. Based on analyzing this model in three popular replica placement policies, we figure out the individual optimal values of these parameters at first, and then work out their optimal combination by GA. Compared with the existing models, our model is easier to solve while reaching more integrative and practical conclusions. These conclusions can effectively instruct the designers to build more reliable large-scale data warehouses.

DBMap: a space-conscious data visualization and knowledge discovery framework for biomedical data warehouse.

Advances in digital imaging modalities as well as other diagnosis and therapeutic techniques have generated a massive amount of diverse data for clinical research. The purpose of this study is to investigate and implement a new intuitive and space-conscious visualization framework, called DBMap, to facilitate efficient multidimensional data visualization and knowledge discovery against the large-scale data warehouses of integrated image and nonimage data. The DBMap framework is built upon the TreeMap concept. TreeMap is a space constrained graphical representation of large hierarchical data sets, mapped to a matrix of rectangles, whose size and color represent interested database fields. It allows the display of a large amount of numerical and categorical information in limited real estate of the computer screen with an intuitive user interface. DBMap has been implemented and integrated into a large brain research data warehouse to support neurologic and neuroradiologic research at the University of California, San Francisco Medical Center. For imaging specialists and clinical researchers, this novel DBMap framework facilitates another way to better explore and classify the hidden knowledge embedded in medical image data warehouses.