A neural network regularization method to address variance inflation in autoencoders

Abstract

There exist various machine learning techniques which can be used to reduce the dimensionality of original data while minimizing the information loss. Principal component analysis (PCA) is one of the most well known such techniques, which transforms the original correlated variables into uncorrelated variables called principal components. Although PCA is known to preserve the total variance of the original data during the transformation, there are some cases with a potential of variance inflation, where the total variance of principal components becomes much larger than that of original variables. It is important to prevent variance inflation, as it can negatively affect the performance of other application systems (e.g. process monitoring systems) which are designed on the basis of principal component with inflated variances. Variance inflation also has a high potential to occur during the training of autoencoder, a special type of neural network performing nonlinear version of PCA. Although there are several neural network regularization methods available to alleviate the problem of variance inflation, none of them is tailored to do such task. To this end, in this work, an alternative neural network regularization method is proposed, which can strongly regulate the total variance in the feature space. Using the Tennessee Eastman process as an illustrative example, the proposed regularization method is compared with the existing ones in terms of neural network overfitting, variance inflation, and training time.