Applying Exponential Family Distribution to Generalized Extreme Learning Machine

Abstract

The learning algorithm of an extreme learning machine (ELM) has two fundamental steps: 1) random nonlinear feature transformation and 2) least squares learning. Since the probabilistic interpretation for a sample by the least squares method follows a Gaussian distribution, there are two limitations in ELM caused by the second step: 1) it may be inaccurate to handle binary classification problems, since the output of a binary dataset has a distribution far from Gaussian and 2) it may have difficulties in dealing with nontraditional data types (such as count data, ordinal data, etc.), which also do not follow Gaussian distribution. In order to solve the above-mentioned problems, this paper proposes a generalized ELM (GELM) framework by applying the exponential family distribution (EFD) to the output layer node of ELM. It simplifies the design of ELM models for task-specific output domains with different data types. We propose a unified learning paradigm for all the models under this GELM framework with different distributions in EFD, and prove that traditional ELM is a special instance of GELM by setting the output distribution as a Gaussian distribution (GELM-Gaussian). We also prove that the training of GELM-Gaussian can be finished in one iteration, in this case, GELM-Gaussian does not slow down the training speed of traditional ELM. Besides, we propose the kernel version of GELM, which can also be concretized to different models by applying different EFDs. Experimental comparisons demonstrate that GELM can give more accurate probabilistic interpretation to binary classification and GELM has a great potential in dealing with a broader range of machine learning tasks.