Handling very large workloads to effectively partition data warehouses: New approach and experimental study

Abstract

A data warehouse (DW) is continuously subjected to large and complex workloads of queries. These queries often require very long response time. To reduce the execution cost of these queries, the DW administrator selects optimization structures as horizontal partitioning which offers performances and manageability advantages without consumption of extra storage space. The partitioning selection problem is known NP-Complete. Several approaches have been proposed in the literature to manage the complexity of the problem and generate valuable partitioning schema. The proposed approaches consider a workload of most frequent queries to optimize. However, the majority of them do not take into account the size of the workload which can be very large. A very large workload can drastically increases the time needed to select partitioning schema and the size of the search space. To tackle this problem, we propose a new approach to handle very large workloads in order to partition effectively a DW. We conduct an experimental study on the ABP-1 benchmark to test the effectiveness and scalability of our approach. We have also conducted validation in Oracle 11g.