Adaptive Noise Level for Stacked Denoising Autoencoder

Abstract

The stacked denoising autoencoder (SDAE) is a slight modification of stacked autoencoder, which is trained to reconstruct a clean version of an input from its corrupted version. It has been successfully used to learn new representations for an unsupervised framework. However, the noise level in the SDAE is determined by experience and remains fixed throughout the training process. To address this limitation, we present an adaptive stacked denoising autoencoder based on the principle of annealing (AdaptiveSDAE), a novel method of adaptively obtaining the noise level. This is achieved by first computing the average noise level for each epoch using a linear average noise level function based on the principle of annealing; and second calculating the noise level for each input neuron based on the average noise level, and the contribution of the input neuron to the activation of hidden neurons (which depend on the input neuron’s value and the weights). Thus, the network includes a combination of features at multiple scales. The experimental results show that our proposed AdaptiveSDAE performed better than SDAE and other unsupervised feature learning methods.