A Hybrid Method for Incomplete Data Imputation

Abstract

With the explosive increase of data volume, the research of data quality and data usability draws extensive attention. In this work, we focus on one aspect of data usability -- incomplete data imputation, and present a novel missing value imputation method using stacked auto-encoder and incremental clustering (SAICI). Specifically, SAICI's functionality rests on four pillars: (i) a distinctive value assigned to impute missing values initially, (ii) the stacked auto-encoder(SAE) applied to locate principal features, (iii) a new incremental clustering utilized to partition incomplete data set, and (iv) the top nearest neighbors' weighted values designed to refill the missing values. Most importantly, stages (ii)~(iv) iterate until convergence condition is satisfied. Experimental results demonstrate that the proposed scheme not only imputes the missing data values effectively, but also has better time performance. Moreover, this work is suitable for distributed data processing framework, which can be applied to the imputation of incomplete big data.