Integration of Data Mining and Statistical Methods for Constructing and Exploring Data Cubes

Abstract

In high dimensional environments, the sheer size and volume of data poses a number of challenges in order to generate meaningful and informative data cubes. Data cube construction and exploration is a manual process in which analysts are required to visually explore the complex cube structure in order to find interesting information. Data cube construction and exploration has been dealt separately in the literature and in the past there has been very limited amount of work done which would guide the data warehouse designers and analysts to automatically construct and intelligently explore the data cubes. In the recent years, the combined use of data mining techniques and statistical methods has shown promising results in discovering knowledge from large and complex datasets. In this chapter, we propose a methodology that utilizes hierarchical clustering along with Principal Component Analysis (PCA) to generate informative data cubes at different levels of data abstraction. Moreover, automatically ranked cube navigational paths are provided by our proposed methods to enhance knowledge discovery from large data cubes. The methodology has been validated using real world dataset taken from UCI machine learning repository and the results show that the proposed approach assists in cube design and intelligent exploration of interesting cube regions.