Supervised Extreme Learning Machine-Based Auto-Encoder for Discriminative Feature Learning

Chi-Man Vong,Chuangquan Chen,Peng Liu,Jie Du,Zhenbao Liu

doi:10.1109/access.2019.2962067

Abstract

In this paper, a Supervised Extreme Learning Machine-based Auto-Encoder (SELM-AE) is proposed for discriminative Feature Learning. Different from traditional ELM-AE (designed based on data information X only), SELM-AE is designed based on both data information X and label information T. In detail, SELM-AE not only minimizes the reconstruction error of input data but also minimizes the intra-class distance and maximizes the inter-class distance in the new feature space. Under this way, the new data representation extracted by proposed SELM-AE is more discriminative than traditional ELM-AE for further classification. Then multiple SELM-AEs are stacked layer by layer to develop a new multi-layer perceptron (MLP) network called ML-SAE-ELM. Benefit from SELM-AE, the proposed ML-SAE-ELM is highly effective on classification than ELM-AE based MLP. Moreover, different from ELM-AE based MLP that requires large number of hidden nodes to achieve satisfactory accuracy, ML-SAE-ELM usually takes very small number of hidden nodes on both feature learning and classification stages to achieve better accuracy, which highly lightens the network memory requirement. The proposed method has been evaluated over 13 benchmark binary and multi-class datasets and one complicated image dataset. As shown in the experimental results, through the visualization of data representation, the proposed SELM-AE extracts more discriminative data representation than ELM-AE. Moreover, the shallow ML-SAE-ELM with smaller hidden nodes achieves higher classification accuracy than hierarchical ELM (a commonly used effective ELM-AE based MLP) on most evaluated datasets.

Highlights

Feature learning or representation learning [1], [2] is a kind of technique that can automatically extract the effective representations needed for classification or other specific tasks from training data
For efficient learning, the optimal output weights for classification in multi-layer Extreme Learning Machine (ELM) (ML-ELM) and hierarchical ELM (H-ELM) are directly calculated by regularized least squares [19] rather than being iteratively updated by gradient descent strategy [20], and the classification error cannot be back-propagated to the stacked extreme learning machine based auto-encoder (ELM-AE)
PRELIMINARIES we briefly review ELM [12], ELM-AE [10] and H-ELM [14]

Summary

INTRODUCTION

Feature learning or representation learning [1], [2] is a kind of technique that can automatically extract the effective representations needed for classification or other specific tasks from training data. Both methods achieve high level representation of input data and outperforms other MLP methods (e.g., Deep Belief Networks (DBN) [16], [17] and Deep Boltzmann Machines (DBM) [18]) In both ML-ELM and H-ELM, the ELM-AEs are not fine-tuned by the classification error when training [10], [14]. For efficient learning, the optimal output weights for classification in ML-ELM and H-ELM are directly calculated by regularized least squares [19] rather than being iteratively updated by gradient descent strategy [20], and the classification error cannot be back-propagated to the stacked ELM-AEs. In other words, the ELM-AEs in ML-ELM and H-ELM extract data representation based on input data X only, while the labels information T is not used.

PRELIMINARIES

ELM-AE

PROPOSED SELM-AE

EXPERIMENTS AND EVALUATIONS

Findings

CONCLUSION