Lightweight Feature Selection Methods Based on Standardized Measure of Dispersion for Mining Big Data

Abstract

Big data analytics is emerging as an important research field nowadays with many technical challenges that confront both commercial IT deployment and big data research communities. One of the inherent problems of big data is the curse of dimensionality. Modern data are described with many attributes and stored with high dimensions. In data analytics, feature selection has been popularly used to lighten the processing load in inducing a data mining model. However, when mining high dimensional data the search space from which an optimal feature subset is to be derived grows exponentially. That leads to an intractable demand in computation. In order to tackle this problem of high-dimensionality and the challenge of achieving high-speed processing over big data, a collection of novel lightweight feature selection methods is proposed in this paper. The feature selection methods are designed particularly for processing high-dimensional data quickly, by fast clustering and separating attributes using the standardized measure of dispersion. For performance evaluation, several types of big data with large degrees of dimensionality are put under test with our new feature selection algorithms. The new methods achieve enhanced classification accuracy within a relatively short time in comparison to existing feature selection methods.